Get documents about "UW Libraries Usability Testing Report - Download as PDF
Document Sample
UW Libraries Website Redesign
Usability Evaluation
Brief Report
January 31, 2009
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
TABLE OF CONTENTS
CONTENTS
Executive Summary ............................................................................................................................................... 3
Introduction........................................................................................................................................................... 4
Methods ................................................................................................................................................................ 7
A. Specified Users ....................................................................................................................................................... 7
B. Specified Goals ...................................................................................................................................................... 8
c. Procedures.............................................................................................................................................................. 9
D. Analysis ................................................................................................................................................................ 10
Specific findings ................................................................................................................................................... 11
A. Look and Feel ...................................................................................................................................................... 11
B. New Design Elements .......................................................................................................................................... 12
C. Basic Tasks ........................................................................................................................................................... 18
D. Conceptual Challenges ........................................................................................................................................ 23
E. Supporting Discovery ........................................................................................................................................... 26
F. UW WorldCat ....................................................................................................................................................... 29
G. Accessibility ......................................................................................................................................................... 32
Conclusions.......................................................................................................................................................... 33
Appendix A: Interview Questions and Task Instructions ..................................................................................... 34
Appendix B: About this Report ............................................................................................................................ 37
Page 2
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
EXECUTIVE SUMMARY
In the autumn of 2008, the University of Washington Libraries launched a redesigned
website. Changes to the site included an updated look and feel, and new design features
intended to improve patron access to resources. To evaluate the effectiveness of the
redesign, a usability test of the new site has now been completed.
Ten users from a range of disciplines and experience levels were asked to perform various
tasks using the new site, and respond to inquiries about their overall reactions to the site.
Their actions and comments were analyzed to determine the effectiveness, efficiency, and
satisfaction with which they were able to accomplish tasks commonly performed by UW
library patrons. During these evaluations, the following findings were noted:
Most users expressed a positive
response to overall site design
and organization
Participants exhibited significant
adoption and use of new
elements including drop down
navigation menus, header search
box, and Qwidget chat link
Participants were easily able to
complete basic tasks such as
looking up books, accessing
Figure 1: UW Libraries Home
Course Reserves, and logging in
Page, September 2008
to their Library Account.
Several usability challenges were also noted, and solutions for some of these have been
implemented. For other issues, further research is necessary:
The Image Collection Search box was misinterpreted by users. It has been removed.
Many users had difficulty navigating the Hours of Operation page. This page has
been revised to more prominently feature web-friendly information.
Users did not identify or utilize the Reference Tools section. Discussions of relabeling
this page or moving its content elsewhere are underway.
The proximity of the “Find It” links and the Worldcat Local “Search Everything” tab
confused users. These two areas have now been visually separated.
Library-provided tools such as Research Help and Subject Guides were underutilized.
The Image Collections and UW Digital Collections content areas were not understood
by users. Navigation and labeling modifications are under discussion.
UW Worldcat search results were too broad and difficult to narrow down for many
users. These findings have been forwarded to the Worldcat Local Team.
Page 3
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
INTRODUCTION
The UW Libraries website serves as a critical access point for materials for both remote and
in-person library patrons. The ability of patrons to easily discover and access resources
through this site is critical to the Libraries‟ mission.
The Spring 2007 UW Libraries Triennial Survey found a sharp increase in remote use of the
Libraries.1 While undergraduate students continued to frequent physical library spaces,
usage by faculty and graduate students had shifted dramatically towards remote access.
Graduate students are now more likely to connect to the library from off-campus than from
any location on campus.2
Another trend noted in the survey data was a growing tendency for students and faculty to
begin their research process on the open internet, rather than the library catalog or other
library-provided discovery tool.
This finding was supported by results
of Project Information Literacy 2008.3
This study conducted discussions with
undergraduate students from many
different campuses, and found that
students often use Wikipedia as a first
step in information discovery. This
project also concluded that the massive
increase in available resources has
paradoxically made research more
difficult for many students.
In October 2007, the UW Libraries Public
Web Operations Group began to evaluate
the UW Libraries website (Figure 2) in Figure 2: UW Libraries Home Page,
2004- August 2008
1
Hiller, Steve. 2007-2008. “Listening to the UW Community: the 2007 UW Libraries Triennial Survey.”
Library Directions: A Newsletter of the University of Washinton Libraries, Autumn-Winter, 17(2).
Retrieved online on January 13, 2009, from:
http://www.lib.washington.edu/about/libdirections/ldfal07-web.pdf
2
Preliminary Selected Results from UW Libraries 2007 Triennial Survey. Retrieved online January 13,
2009 from: http://www.lib.washington.edu/assessment/surveys/survey2007/documents/ARLForum
TriResults2007.pdf
3
Project Information Literacy 2008 video. Retrieved online on January 23, 2009 from:
http://www.youtube.com/watch?v=RKI7yOl8nQY
Page 4
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
response to these changes in patron behavior, and in recognition of the increasing
importance of online access. The site was evaluated based on its ability to meet user needs
and to further the Libraries‟ goals, which include being an “Anytime, Anyplace” library, and
enabling users to function self-sufficiently on their own terms.
The Public Web Operations group found that improvements could be made to the site to
better serve these goals, and began a data-driven design process to improve this critical
access point.
GOALS OF THE REDESIGN
Based on an analysis of user behaviors outlined in the Triennial Survey and with the overall
strategic goals of the UW Libraries in mind, the Public Web Operations Group developed the
following high level goals for a redesigned website:
Extend navigation and access to critical resources across subpages
Increase access to tools and resources that help users be self-sufficient
Reduce redundant information
Preserve easy access to existing resources
Improve the accessibility of the UW Libraries Website
Update look and feel to be more closely aligned with the main University website
To inform the implementation of these goals in specific sections of the site, patron behavior
was analyzed using a range of data sources, such as the previously mentioned Spring 2007
UW Libraries Triennial Survey, and a series of focus groups conducted in November 2007 to
obtain information about user preferences.
Based on these findings, two early prototypes were developed (Figure 3).
Figure 3: Early Design Prototypes
Page 5
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
Redesign Timeline
These prototypes were evaluated in a
preliminary round of user interviews in May
2008.
2007
Next, user feedback about the advantages Spring Triennial Survey
and disadvantages of those early prototypes
was used to develop a single design.
November Focus Groups
The final layout included significant
revisions to the home page and to the
2008
headers, footers, and navigation bars of
May/June User Evaluation of
subpages. This redesign did not directly
Prototypes
include other services linked from the
website (such as WorldCat Local and
Course Reserves).
September New Site Launch
The redesigned site was launched in
September 2008, and the focus of this November Usability Testing
report is the usability testing of that
release, which was conducted in November December- Revisions based on
2008. Present Usability Findings
This testing effort aimed to evaluate the
effectiveness and efficiency of the design
for patrons who use the UW Libraries
website. Figure 4: Redesign Timeline
The techniques and methods used in this study were designed to measure usability, as it is
defined in the International Standard of Usability:
“Usability: the extent to which a product can be used by specified users to
achieve specified goals with effectiveness, efficiency and satisfaction in a
specified context of use.” 1
Accordingly, the study was structured around three main areas: a group of specified
users, their specific goals, and analysis of their satisfaction and success in achieving
those goals with effectiveness and efficiency.
1
International Standard of Usability, ISO 9241-11: Guidance on Usability, retrieved online on
1/15/2009 from http://www.usabilitynet.org/tools/r_international.htm#9241-11
Page 6
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
METHODS
A. SPECIFIED USERS
The specified users for this investigation were undergraduate students, graduate students,
and faculty of the University of Washington. A target number of participants for each user
group was set based on relative proportions of the University population,1 as shown in
Figure 5. In keeping with the real-world populations, the majority of participants in this
study were undergraduates, with significant representation from graduate students and
faculty members. Ten participants were recruited from a variety of disciplines and
experience levels.
80
70
Percentage of Population
60
50
40
30
20
10
0
Undergraduate Graduate & Faculty Library
Professional Employees
Figure 5: 2007 Population Distribution at the University of Washington
1
Demographic statistics are taken from the UW Libraries report to the Association of Research
Libraries for 2007, retrieved on 1/15/2009 from:
http://www.lib.washington.edu/assessment/stats/default.html
Page 7
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
B. SPECIFIED GOALS
The particular user goals measured in this study were based on previous research data
which described common actions of users of the UW Libraries. These goals included
discovering, locating, and using library resources. For the purposes of this study, users‟
goals were approximated with a diverse set of basic and advanced tasks, which participants
were asked to complete using the new site.
Tasks were designed to both simulate common goals of patrons (such as finding books) and
to evaluate the particular goals of this redesign (such as improving navigation across
subpages).
Both the wording of the task instructions, and the structure and order of the script, were
used to maximize evaluation potential. For example, participants began their session at the
Odegaard Undergraduate Library Home page, rather than the main UW Libraries Home
page, and were instructed to accomplish the first three tasks from this page. After
completing those tasks, participants were directed to the main UW Libraries Home page for
the remainder of the session.
This structure allowed the
evaluation of design elements Usability Evaluation Tasks
such as the persistent navigation
drop-down menus and the catalog From the Odegaard Library Home Page
search box in the header. Unit: Find a specific book
Unit: Access your library account
In the actual study, each task Unit: Find an article about a general topic
was presented to users with a
brief narrative providing From the Main Home Page
instructions and context for the Main: Get help with Research
task. The full text of the task Off Campus Access to restricted resources
instructions is available in Subject Guides: explore a new subject area
Appendix A. Find an image of a specific person
Look up a specific library policy
Find a specific book
Pay a fine
Locate Course Reserves
Look up a specific fact (Reference Tools)
Reserve a study room
Request that the libraries purchase a book
Ask a librarian for help
Look up the opening hours of a library location
Page 8
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
C. PROCEDURES
STUDY PROTOCOL
Usability sessions were facilitated by a Systems Librarian or by a Graduate Staff Assistant in
the IT Services Department of the University of Washington Libraries.
Each session was conducted according to a strict protocol including a written script
for the facilitator, to ensure consistency. Participants were encouraged to provide
both positive and negative feedback and assured that the evaluation was not
intended to measure their abilities, but rather to test the site itself. Participants
were asked to complete a series of 15 tasks using the UW Libraries website. During
the tasks, the facilitator prompted participants to describe their thoughts processes
and provided assistance with task instructions but not with execution. Once all tasks
were completed, the testing software presented the participant with an on-screen
survey to evaluate their experience with the site.
EQUIPMENT
The usability tests were performed in a dedicated usability lab located within Suzzallo
Library. The room contains a desk with a computer, monitor and chair. An additional chair
was used by the testing facilitator, who was seated next to the participant.
This machine was running Microsoft Windows XP Professional Version 2002, Service Pack 3,
Internet Explorer 7, and Mozilla Firefox 2.0.
Morae 3, a commercial user testing software system made by Techsmith
(http://www.techsmith.com/morae), was used to present task instructions to participants,
and to record their screen actions, comments, facial expressions, and survey responses.
To facilitate efficient session administration and results analysis, the „autopilot‟ feature of
Morae 3 was used, which generated popup instructions for each task.
The autopilot mode automatically logged the beginning and ending of each task based on
when the participant pressed the „start‟ button on the task instruction window. Once the
sessions were recorded and analyzed, Morae‟s built in graphing function allowed the quick
production of quantitative results displays.
Page 9
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
D. ANALYSIS
Participants‟ actions and comments were evaluated using a variety of methods to measure
the effectiveness and efficiency with which they were able to accomplish their goals, and
their overall satisfaction with the site.
SATISFACTION
Participant satisfaction with the web experience was evaluated by examining the comments
users made both verbally during the session and in response to a written survey at the end
of the session.
EFFECTIVENESS
The effectiveness with which participants completed tasks was analyzed by scoring the
difficulty level of each task for each user. Difficulty scores were assigned by analyzing
session recordings to determine whether each task was completed with ease, completed
with difficulty, or failed to complete.
For some tasks, participants did not use the resource provided by the library website and
instead used an alternate strategy. For example, in several cases participants stated that
they would ask a librarian in person rather than use the website. These tasks were scored
as failed to complete; although the alternate strategy may have been successful in
accomplishing the user‟s goal, their rejection of resources prepared by the library indicates
a potential problem with those areas of the site.
EFFICIENCY
Because of the occurrence of „alternate strategy‟ tasks, measurement of Time on Task was
not found to be an accurate indicator of efficiency. Several participants spent very little time
on tasks for which they used an alternate strategy, however the low task time did not
indicate efficient task completion.
Instead, both the difficulty score and a detailed summary of participant actions and
comments were used to evaluate efficiency. This summary was compiled from the session
recordings and sorted by task, to identify inefficiencies that arose for multiple users.
Page 10
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
SPECIFIC FINDINGS
A. LOOK AND FEEL
Most of the study participants found the site easy to use, with eight out of ten participants
rating the site‟s “ease of use” highly (four on a scale of one to five). Most users also
displayed a positive response to the new visual design of the site, with some dissenting
viewpoints. The doctoral candidate and faculty member participants commented that the
pages had too much content:
“Screens are too busy…too much on a website makes too much to parse at once
and I tend to get frustrated.”
However the majority of study participants, including all of the undergraduate students and
one master‟s student, felt the new design was easy to navigate:
“I liked the simplicity, the ease to access the most commonly asked information,
and the prominent search box.”
“I liked how everything was laid out, under broad but defined categories that you
would expect information to be under.”
Based on focus group comments, some goals of this redesign included creating an
environment that felt more active and inviting to users, and was less reliant on lists of text
links. The user data collected in focus groups and preliminary testing, in which users
responded positively to the stronger color scheme with UW branding, was supported by this
round of testing, although once again a disconnect was apparent between the perceptions of
undergraduate students and faculty/graduate students. Faculty and graduate students
tended to prefer a more subdued style, commenting:
“The site is very cluttered (could be neater).”
“The color gradations -- purple to white -- are very annoying and distracting.”
Undergraduate students commented approvingly on the layout, or expressed a wish for it to
be even more dynamic:
“I liked how the site wasn't crowded or busy, but was clustered in sections.”
“Also the site looks a little dull. It needs maybe some flash enabled things. “
Page 11
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
B. NEW DESIGN ELEMENTS
Several tasks were planned specifically to evaluate the use and efficacy of new structural
and visual elements. These new elements included a persistent navigation bar with drop-
down menus, changes to the catalog search presentation, and a Qwidget chat link with a
librarian‟s image. All of these new elements saw significant adoption and use by
participants in this study.
DROP-DOWN NAVIGATION
The previous version of the libraries website included a persistent navigation bar, as shown
in Figure 6. This bar included links to the main areas of the site and appeared on most
subpages.
Figure 6: Persistent Navigation Bar
In the new design, the navigation bar has been modified to include drop-down menus for
each area of the site (see Figure 7). This navigation structure allows users to jump directly
between site subpages, without needing to return to the home page.
Figure 7: New Drop-Down Menus
Page 12
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
As shown in Figure 8, most users took advantage of the drop down menus at some point
during the evaluation. For some tasks they chose to return to the home page to scan all
available links, however eight out of ten participants used the drop down menus at least
once.
Drop-Down Menu Use Across All Tasks
18
16
14
12
10
8
6
4
2
0
Figure 8: Drop-down Menu Use
Page 13
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
PRESENTATION OF SEARCH
Focus group comments
revealed user confusion about
the multiple search boxes on
the home page (Figure 9). The
subdued color scheme and
placement of the main catalog
search box was also noted as
a challenge by some users.
To address these issues, the
main catalog search box was
moved in this design to a
more central location and
strongly identified with color
and prominent tabs (as in Figure 9: Two Search Boxes
Figure 10). The “site search”
box has also been removed
from the home page.
Eight out of ten users
identified the catalog search
box as the first or second item
they noticed on the home
page. Searching the library
catalog is one of the most
common reasons patrons visit
the UW libraries website, and
making this search easy to
locate strongly supports
efficiency for users. In their
comments, five users
specifically mentioned the
prominent search box with
tabs as one of the features Figure 10: Prominent Catalog Search
they liked most about the site.
Page 14
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
A catalog search box has also been added to the header bar of subpages, to facilitate
searching from anywhere on the site, as shown in Figure 11.
Figure 11: Subpage Header
To evaluate the effectiveness of this new presentation of catalog search, each usability
session began at the Unit Homepage for the Odegaard Undergraduate Library. The first
task instructed participants to find a book, and all ten users quickly located the new catalog
search box in the website header, indicating that this feature supports user efficiency.
Two users chose to limit their search to the UW-only catalog, and a third participant
expressed confusion about whether the box would search UW Worldcat or the UW-only
Catalog. These actions supported previous findings that some patrons prefer not to see
results from other libraries. See the UW Worldcat section of this report for further
discussion.
REVISIONS: HEADER SEARCH
To increase clarity about the function
of the Header search box and support
different searching preferences, a radio
selector has been introduced that allows
patrons to select UW Worldcat,
UW-Only Catalog, or Site Search. Figure 12: Revised Header Search
Page 15
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
QWIDGET CHAT LINK
With this design, the main „Ask Us‟ link
has been moved to a more prominent
location on the home page, and a new
„Qwidget‟ chat link (Figure 13) was added
to the home page. The Qwidget is an
image link that resembles an instant
messaging program screen.
The Qwidget and „Ask Us‟ links offer
similar but not identical functionality.
Clicking on the Qwidget link that shows
an image of a librarian pops up a small
chat window; this service is only
available during normal UW Libraries
operating hours.
The „Ask Us‟ link, in contrast, offers
patrons a choice of whether to ask their
question via chat, email, or telephone.
The „Ask Us‟ chat link opens a full new
browser page that requires visitors to Figure 13: Qwidget and Ask Us Chat Links
enter their name and email address, and
provides detailed instructions about how to effectively use the chat software. This chat
format is also connected to a nationwide collaborative reference service that is available
twenty-four hours a day, seven days a week.
Chat reference staff reported significantly increased volume of chat questions after the
release of the new home page design, and an analysis of chat logs confirmed that 145%
more chats were initiated by users during 7 days in the Autumn 2008 quarter, as compared
to the same week in 2007. Clearly the new prominent placement of the Ask Us and Qwidget
chat links has been successful in promoting this resource to patrons; one participant in this
study commented, “I also liked the chat feature because it makes me feel connected to the library. It is
like a lifeline.”
One goal of this usability evaluation was to explore the dynamics of user behavior around
the „Ask Us‟ and Qwidget chat links. The chat log analysis also found that about 30% of
chats in 2008 originated from the Qwidget link, which matched the user behavior observed
in this study. All the participants in this evaluation successfully located a link to chat with a
librarian; four out of ten used the Qwidget link.
To better understand user preferences between these two options, participants were asked
about their decision and what motivated it. Some users commented that they noticed the
Qwidget link because it contained an image, and clicked on it because it seemed more
immediate, while one stated that she did not use the Qwidget because she assumed the
image was an advertisement. Several users also stated that they preferred the „Ask Us‟ link
Page 16
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
because they wanted the option to email a librarian rather than chat online, and one user
mentioned that he remembered a librarian telling him that the Qwidget was only available
during regular hours.
The chat statistics and user comments from these evaluations provide mixed results about
the efficacy of the Qwidget link. With a third of patrons choosing this option, the Qwidget is
clearly a powerful way of connecting with a large user group. The immediacy and casual
format which are attractive to some users, however, appears to be a deterrent for others.
FUTURE DIRECTIONS: QWIDGET CHAT
For the time being, both the „Ask Us‟ and the Qwidget chat link will remain on the
homepage, and dissemination of the Qwidget link to subpages will be encouraged.
More research in future projects will help us better understand how people use and
understand the different functionality of the Qwidget and Chat links.
Page 17
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
C. BASIC TASKS
In general, participants were able to easily locate major resources and perform common
tasks such as looking up books and accessing their library accounts. Of the sixteen tasks
that users were asked to perform, results analysis showed that eight tasks presented little
or no difficulty for a large majority of users. Effective and efficient completion of these basic
tasks appears to be well-supported by the current design.
Average Difficulty by Task
Failed to 2
Complete 1.8
1.6
1.4
Completed 1.2
with Difficulty 1
0.8
0.6
0.4
0.2
0
Completed
with Ease
Figure 14: Average Difficulty by Task
Several of the tasks participants were asked to perform mimicked basic tasks such as
looking up books and accessing their personal account information.
Participants in this study were able to complete most of these functions with little to no
difficulty. (See the Methods section for a description of the difficulty scoring procedure).
Page 18
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
Tasks which presented little or no difficulty included:
FIND A BOOK OR ARTICLE
These tasks were completed with ease by most participants from both the Unit page and the
main home page. Most users quickly located the catalog search. Some users, in particular
those with less library experience, had trouble identifying appropriate article databases and
navigating through the database search process. The increasing availability of articles
through WorldCat should prove useful for this type of user.
ACCESS LIBRARY ACCOUNT
All participants found this task easy. Most immediately identified the link in the blue
navigation bar. One user mentioned that she would expect her library account information
to appear in the MyUW interface. This integration is not supported by the current system,
but may be useful to consider in the future.
PAY A FINE
This function was completed with ease by all participants, except one who simply stated
that he would not perform this action online.
ASK FOR HELP
Nine of ten participants easily located the links to ask for help online, although several
stated that they would prefer to visit a library in person to ask for help.
OFF-CAMPUS ACCESS
Although this evaluation was conducted on campus, in one task users were instructed to act
as though they were attempting to access a resource from off-campus. Nine out of ten
participants completed this task with ease; even participants who stated they had never
used the off-campus access link before were able to quickly scan the page and find the link.
An analysis of online help queries received by librarians revealed that many of those
requests relate to difficulties with off-campus access. Based on findings in this study,
problems with off-campus access do not stem from the placement of the link on the home
page. Rather they are likely to be related to technical functioning problems, such as browser
settings and cookies. Another source of difficulty is the non-specific error messages users
receive when attempting to access a restricted resource without first logging in as an off-
campus user. Such messages are generated by the database vendor and often simply state
that access is denied, without instructions about how to log in to off-campus access.
Unfortunately the libraries cannot modify the user‟s browser settings nor the vendor‟s error
message content.
FUTURE DIRECTIONS: OFF-CAMPUS ACCESS
The UW Libraries has limited control over some common causes of problems with off-
campus access. Given the heavy use of this function and the frequency of difficulties,
however, future evaluations and analysis of potential mitigation strategies is warranted.
Page 19
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
RESERVE A STUDY ROOM
For this evaluation, participants were asked to locate the section of the site which allows
patrons to reserve a study room. All participants completed this task with ease. The
system for reserving a study room in the libraries is known to be difficult to navigate,
provide little visibility of available times, and little feedback about user actions. Further
examination of this system is warranted, but outside the scope of this project.
User actions with some other basic functions, such as searching for an image and looking up
the library‟s opening hours, revealed some non-intuitive structures that were presenting
challenges to patrons. Although many participants were still able to effectively accomplish
their tasks, the efficiency with which they acted was decreased by these design elements.
COURSE RESERVES
Nine out of 10
participants easily located
the link for Course
Reserves, including
several who stated they
had never previously
used that section of the
site. One participant, a
novice user of the site,
experienced significant
difficulty and was not able
to locate Course Reserves
at all.
That particular user did
not maximize the browser
window, and did not scroll
down to discover the
Course Reserves link in
the “Using the Libraries”
section. She was also
unfamiliar with the term
“Course Reserves.”
Figure 15: Minimized View of Home Page
Page 20
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
FUTURE DIRECTIONS: COURSE RESERVES
Although the average difficulty rating for this task was very low, given the importance of
Course Reserves to undergraduate students, a more prominent location for this particular
link may be justified. Including this resource within the “Find It” section to ensure higher
visibility is under discussion by the Public Web Operations Group. Future projects should
also examine whether patrons recognize and understand the term “Course Reserves.”
IMAGE SEARCH
Eight of ten participants experienced
difficulty with, or failed to complete,
the image search task. Most began by
stating that they would use Google to
look for an image, rather than the
Libraries website.
With this redesign of the UW Libraries‟
website, the Image Collections link has
been added to the main home page.
Seven users successfully located this
link and navigated to the Image
Collections page.
Once users arrived at the Image
Collections page, however, it became
clear that the search box on the page
did not correspond to most users‟
mental models of search. Six out of
seven attempted to search for a key
word relating to the image they were
seeking, when in fact the search
function only allowed searching for a Figure 16: Image Collection Search
particular Image Collection by name.
REVISIONS: IMAGE SEARCH
Since those who already know the name of the collection they are seeking can easily
browse for it, the collection search box has been removed to help users form a better
mental model of the Image Collections page, and to facilitate browsing by novice users.
Page 21
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
LIBRARY HOURS
A strong majority of users were easily able to look up information about the libraries‟ hours
of operation. Observations of user behavior during this task, however, revealed a design
challenge of the existing layout that impacted the efficiency with which users accomplished
this task.
Both a web-formatted listing
and a printer-friendly
version of the hours for
different libraries are
available to library patrons.
As shown in Figure 17, links
to these versions were
displayed on the same page
with the print version on
top. Many users
immediately clicked on the
print version, even if they
intended only to read the
information online. This led
to a delay while the
document loaded, followed
by the visual challenge of
scanning a document on
which information was
tightly condensed to fit on a
single printed page.
Figure 17: Library Hours Display
REVISIONS: LIBRARIES AND HOURS
To alleviate this difficulty, the
“Libraries and Hours” section
has now been reformatted.
The links to printer-friendly
formatted documents are
now listed in the right
sidebar, where they are
easily spotted by those
seeking the printable version,
but will not distract users
seeking web-formatted
information.
Figure 18: Revised Library Hours
Page 22
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
D. CONCEPTUAL CHALLENGES
Over the course of these evaluation sessions, a number of trends in user behavior emerged
that revealed confusing page structures and labels, and severely impacted users‟ ability to
effectively accomplish their goals using this website.
REFERENCE TOOLS
By and large, the Reference Tools section was overlooked by participants, even when they
were specifically seeking reference facts. Eight out of ten participants failed to complete the
reference task. Most stated that they would normally search on Google or Wikipedia, rather
than use the library. When prompted to search on the libraries website, most users tried a
keyword search in the library catalog or an article database. Only one out of ten users
chose the “Reference Tools” link on the home page. When participants were asked about
their actions and choices, several stated that they did not know what the term „Reference
Tools‟ meant. Even frequent library users who had been trained in research techniques did
not visit this section. One user stated that she did not expect the library to carry such basic
resources as encyclopedias; another stated that she would look in the physical reference
section of the Library for materials such as encyclopedias, but it did not occur to her to look
for them online. Taken together, these findings indicate that many patrons no longer have
a concept of “Reference” and that labeling these resources as “Reference” may actually
prevent users from discovering them.
The user behavior observed in this study, however, does not correspond with web traffic
records. During the Autumn of 2008, the Reference Tools page was consistently among the
top ten most visited pages on the libraries website, and was accessed even more frequently
than the operating hours page. Several factors mitigate the impact of these statistics; for
example, a page template used frequently on the main University of Washington site
includes a link to the Libraries‟ “Reference Tools.” The widespread use of that template may
lead to accidental traffic that inflates the page visit statistics. It is also possible that a very
small number of users (for example, Library staff) visit the Reference Tools section
frequently.
FUTURE DIRECTIONS: REFERENCE TOOLS
At present this discrepancy is not fully understood. Members of the Browsable Resource
Discovery (BRD) Group, which is responsible for the content of the Reference Tools
Section, are currently evaluating this section to determine its relevance and
effectiveness. Some resources in this section are duplicates of information found in other
areas of the site, and will be integrated with those listings. BRD is also considering
strategies to improve user discovery of these resources, perhaps by renaming the section
or merging it with another section more likely to be explored by patrons.
Page 23
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
SEARCH EVERYTHING AND “FIND IT”
In this design, the UW Worldcat search box and tabs were prominently placed in the center
of the page. The “Find It” section, which lists article databases and other electronic
collections and resources, is also featured near the top of the page.
All participants used the catalog search, but over the course of the evaluation sessions it
became clear that some participants avoided the “Find It” section of the page entirely.
When asked, several stated that they believed the “Search Everything” button in the catalog
search would do just that – search through not just books in the catalog but all available
library resources, including all article databases and other collections.
Figure 19: Layout of Find It and Search Everything
Page 24
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
REVISIONS: SEARCH EVERYTHING AND “FIND IT”
While the UW Worldcat search does access a wide variety of resources, including some
images and articles, it does not currently search all available library materials. To
combat this misapprehension, the page layout and labeling has been slightly modified to
increase the visual distinction between these two sections of the site.
A dividing line has been placed between the two sections, and the “Search Everything”
tab has been renamed and moved so that it appears after the Books, DVDs, CDs, and
Articles tabs of the UW WorldCat search module. This placement increases the distance
of the label from the “Find It” section. The ordering of the “Search All” tab at the end of
the row of other tabs also creates a visual implication that the “All” refers specifically to
the preceding four tabs.
Figure 20: Revised “Find It” and UW Worldcat Tabs
Page 25
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
DIGITAL COLLECTIONS AND IMAGE COLLECTIONS
Some user confusion during the image searching task, and a general avoidance of the “Find
It” section, led us to look more closely at how participants understood that section. When
asked about what they thought the different link names meant, most clearly understood the
contents of “Articles and Databases,” “Electronic Journals,” and “Electronic Newspapers.”
However several users expressed confusion over the distinction between Image Collections
and Digital collections. Even when they followed the links and looked at those sections,
they found that many of the materials in Digital Collections were in fact images.
Another question to consider is the placement of Special Collections on the Libraries
website. Special Collections overlaps significantly but not completely with the Digital
Collections site, but the link to Special Collections is currently located in a different section
of the site (in the “About the Libraries” section, under “Collections”).
FUTURE DIRECTIONS: DIGITAL COLLECTIONS AND IMAGE COLLECTIONS
There are important differences between these content areas, and the best way of
presenting them transparently to patrons in a way that is both accurate and inviting
remains an open question. No specific changes are currently planned for these areas, but
the UW Libraries Public Web Operations Group will be examining them more closely in
future discussions.
E. SUPPORTING DISCOVERY
A major goal of this redesign, and of the UW Libraries in general, is supporting patron
discovery of resources. Focus groups and the Triennial Survey both found that users were
likely to look outside the library (to resources such as Google and Wikipedia) for subject
discovery searching. Observations of user behavior in these sessions confirmed those
findings.
SITE SEARCH AND BROWSE
With this design, the site search box was removed in order to alleviate user confusion about
multiple search boxes, and instead of a search box on the main page, a link to Site Search
was placed in the page footer.
Page 26
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
The task which most closely addressed this aspect of the site instructed participants to
locate a specific Libraries‟ policy (relating to whether non-affiliated visitors are allowed to
use Library materials). Most users encountered significant difficulty with this task, and
several were not able to successfully locate the section of the site which explicitly states the
policy.
No participants used the Site Search footer link to look for this policy, and their efforts to
browse were hampered by overlapping content in several different sections of the site. In a
survey distributed shortly after the redesign was implemented, one user commented, “Site
search is so small I overlooked it and had trouble finding the information until someone
showed me.” During this study, one participant specifically mentioned that she wished there
was a site search feature.
REVISIONS: SITE SEARCH AND BROWSE
To address these difficulties, a site search option has been added to the search box in the
standard page header (as shown previously in Figure 10).
FUTURE DIRECTIONS: SITE SEARCH AND BROWSE
A proposal to inventory the site content to reduce duplication and create an A-Z site
resource list is under discussion.
SUBJECT GUIDES
Nine out of ten participants did not use the Subject Guides when prompted to look for
resources in a new domain; most stated that they would use Google, or when prompted to
use the Libraries website, performed a keyword search in the catalog.
Participants who attempted to use the Subject Guides for other tasks experienced difficulty
in matching the specific nature of their task to the appropriate broad subject domain. Other
participants, when asked, explained that they did not find the Subject Guides useful
because the guides were too broad, and did not directly apply to their subspecialty.
FUTURE DIRECTIONS: MENTAL MODELS
The Public Web Operations Group is considering a generative research project to help
build a mental map for how UW Libraries patrons approach their research tasks. Such a
model may provide more insight into how to best present content that will correspond
closely with specific user goals and tasks.
Page 27
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
RESEARCH HELP
Another underutilized area of the site was Research Help. When instructed to seek general
assistance with a research project, eight out of ten participants did not locate or use this
section. Many stated they would pursue alternate strategies, such as visiting a librarian in
person.
FUTURE DIRECTIONS: RESEARCH HELP
The Research Help Section is currently a work in progress. An evaluation of this area of
the site is underway to eliminate dated or irrelevant content, condense quick “how-to”
instructions, and create richer instructional content around key areas, including short
tutorial videos. A new platform for delivery is also being explored to provide more
interaction with the video content, such as ratings and comments. Another goal for the
Research Help Section is to support links from different content areas of the larger site,
which can provide users with context-sensitive help.
BRANCH LIBRARY LISTINGS
While several participants did make use of the top black bar which contains links to all the
different physical locations in the UW Libraries system, comments from patrons immediately
after the new site was launched and in particular from users of the Health Sciences library
indicated patron difficulty in locating those links.
It is difficult to determine whether the problems reported with finding the new location of
Branch Library links are a consequence of a lack of familiarity with the new site design, or
are truly inherent to the new placement and coloring. The Public Web Operations group will
revisit this issue in the future to confirm whether findability of Branch Library listings is an
ongoing problem.
Page 28
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
F. UW WORLDCAT
Over the course of the evaluation sessions, several issues relating to the UW WorldCat
catalog search were noted. Although the functioning of UW WorldCat is not within the scope
of this study, the findings described below have been noted and forwarded to the WorldCat
Local team.
NAMING
Currently, the UW Libraries website supports searching with either UW Worldcat or with the
traditional UW-only library catalog. The UW Worldcat search box is featured on the home
page as the default search engine and the UW-only catalog is accessible through a link
beneath the main search box. When this study began, the UW Worldcat search box was
labeled only with the phrase, “search the UW libraries and beyond.” User feedback in this
study and in previous research indicated confusion about the lack of a name on the UW
Worldcat search box, and doubt about what exactly that module would search.
REVISIONS: UW WORLDCAT NAMING
To help users better
understand their
catalog searching
options, the label „UW
Worldcat‟ and an
information icon link
to provide further
details about the
catalog search were
added to the UW
Worldcat search
module. Figure 21: Revised UW Worldcat Search Label
Page 29
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
HOME LINK
The “home” link within the UW Worldcat catalog search was misused. Several users clicked
on this link, intending to return to the main UW Libraries Home Page, but were instead
directed to the UW Worldcat Catalog Search home page.
DISSATISFACTION WITH SEARCH RESULTS
The UW Worldcat search was heavily used by participants in this study, and they expressed
appreciation for the tabbed search interface and easily accessible results filters. Four out of
the ten participants, however, expressed dissatisfaction with the UW Worldcat search
results, primarily because they were not limited to resources available on campus:
“I disliked the fact when you use the search feature on the front page it searches the
whole world and not just UW. I'm not going to go around the world for a book. “
Although this study sample represents only a small number of users, this finding echoes
comments made by users during other evaluations. During the design prototype evaluations
conducted in May 2008, five out of ten participants expressed confusion at the volume of
UW Worldcat search results and a desire to have their results default to display only
materials available on the UW campus, or to have the UW-only catalog as the default search
engine.
Even users who were aware that they could limit their results to UW-only often still
expressed dissatisfaction with the volume and origin of UW Worldcat results. Other users
expressed confusion about exactly what the UW Worldcat system searched, and reluctance
to use such a large system because of a belief that larger databases are more prone to
error.
PROBLEMS WITH LIMITING SEARCHES
Two of the three users who
chose to search within the
UW Worldcat for an image
experienced difficulties
relating to restricting their
search to visual materials.
Participants were instructed
to search for a picture of
Angeline, the daughter of
Chief Seattle. The UW
Libraries owns several
images of this historical
figure, which are easily
retrieved from the UW
Worldcat via a keyword
search (Figure 24). Figure 22: Worldcat Image Search Results
Page 30
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
However if those results are limited to “Visual Materials” or if the Advanced Search feature
is used and the initial search is restricted to Visual Materials Only, none of the UW Holdings
are retrieved and only images held by other libraries are returned (Figure 25).
Figure 23: Worldcat Image Search Results – Limited to Visual Materials
Similarly, a keyword search for the book, “The Devil‟s Highway,” returns that item as the
second result (Figure 26).
Figure 24: UW Worldcat Keyword Search Results
Page 31
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
However a title search for the same book does not return the item at all, as shown in Figure
27. Users found it difficult to interpret such results and were often led to incorrectly
conclude that the UW did not own a certain item. One user commented:
“Sometimes it didn't limit the media quite like I'd expected it to--i.e. I wanted it to return
'books' and the first five (or so) results were articles in a book. This was particularly
confusing when I was doing a title search for a book--Had I entered the title wrong? It
was odd!”
Figure 25: Worldcat Title Search Results
G. ACCESSIBILITY
While not yet tested formally with users, accessibility considerations were at the forethought
of the new design. Table-based layouts were replaced in favor of standards-based
design. Features like "skip to content" links, alternate image text, video transcriptions, and
semantic mark-up were implemented to help improve access to content, especially for users
who experience the site with assistive technologies. Also, the site now makes extensive use
of javascript to provide more dynamic access, such as flydown menus and tabbed
searching. These enhancements have been implemented according to „progressive
enhancement‟ techniques, which improve functionality for browsers that are capable of
handling javascript and hide it from browsers that are not.
Page 32
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
CONCLUSIONS
Overall, the changes introduced with the new design of the UW Libraries website have been
found to improve access and support user efficiency. Notably, the online help chat link and
the new drop-down navigation menus have seen immediate adoption and heavy use.
This study also highlighted several remaining usability challenges; as noted in the Specific
Findings section of this report, some of these challenges are simple and have already been
addressed. For example the placement of different formats of information about the
Libraries operating hours has been modified, the Worldcat Search module has been clearly
labeled, and the misleading Image Collection Search box has been removed.
Other issues which emerged from these evaluations are more complex and will be the focus
of future research by the Public Web Operations Group and other stakeholders. Priorities for
closer examination include:
Evaluating the content and labeling of the Reference Tools section
Developing a strategy for the effective presentation of the Digital Collections and
Image Collections
Researching patron‟s mental models to better align web site content with user goals
Conducting a content inventory to eliminate duplication and create structures to
better support user browsing
Continuing to develop the Research Help section of the site to support interaction
and contextual assistance
Page 33
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
APPENDIX A: INTERVIEW QUESTIONS AND TASK INSTRUCTIONS
SHORT INTERVIEW
Before we get started, I‟d like to ask you a couple of questions about yourself:
1. Which academic department are you affiliated with?
2. What is your year in school?
3. Is there a project or assignment you are working on right now?
4. When you need to find information for a class paper or project, where do you usually
go?
5. When you use the libraries website, do you usually start from the main libraries
home page or from one of the unit home pages, like Odegaard?
6. What kinds of things do you normally do when you visit that page?
7. How often do you use the library?
8. Have you ever been to class about how to use the library?
TASK INSTRUCTIONS
Unit Homepage Tasks (participants started from the Odegaard Home Page)
1. Unit: Find a Book
Find the book, “The Devil‟s Highway” by Luis Alberto Urrea.
2. Unit: My Account
Please return to the page that you started from. You need to go check out the
book you just found, but first you want to see if you have any other books
due back soon, so you can return them on the same trip. Check to see if you
have any books due soon.
3. Unit: Find Article
Please return to the page that you started from. Find an article about
immigration and the United States.
Page 34
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
Main Homepage Tasks (participants started from the Main home page)
4. Research Help
a. You‟ve been assigned a research paper, but have never used the library
before and have no idea where to begin. What would you do?
b. Faculty/Grad Student version: You’ve assigned your students a research
paper, and you know that your students are going to struggle with figuring
out how to correctly cite their sources. Where would you direct them for help
with that?
5. Off Campus Access
Some library materials are free for anyone, but some are only for use by UW
students and faculty. If you‟re off campus and want to use some of the
restricted resources, how do you access them?
6. Subject Guides
a. You‟re taking an introductory course in psychology and need to find
information for a research paper. Find out what resources are available for
psychology research.
b. Faculty/Grad Student version: You’re teaching an introductory class in your
department and are assigning your students a research paper, but your
students won’t know how to begin their research. Where would you tell them
to look?
7. Find an Image
Find a picture of Angeline, the daughter of Chief Seattle.
8. Borrowing Privileges
a. You're taking a quarter off and are not sure whether you'll be able to come
and use the library while you are not registered. Find the library's policy for
letting nonstudents check out books.
b. Faculty/Grad Student version: You're taking a leave of absence and are not
sure whether you'll be able to come and use the library while you are on
leave. Find the library's policy for letting non-affiliated people check out
books.
9. Find a book
A friend recommended you read the book, „The World is Flat‟ by Thomas
Friedman. Find out if the library has a copy of that book that you can borrow.
10.Pay a Fine
You checked out „The World is Flat‟ but forgot to return it on time and now
have a fine. Where would you go to pay the overdue fine?
Page 35
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
11.Course Reserves
a. Your Russian History professor, Barbara Henry, put some articles on course
reserves and told you to get them from the library. Look up the course
reserves for Russian Culture and Civilization -- RUSS 110.
b. Faculty/Grad Student version: You are setting up a course for next quarter.
Where would you go to put some materials on electronic reserve for the
students to access?
12.Reference Tools
a. Please go to the UW Libraries main home page. You need to look up the
population of Russia for your first paper, but you‟re not allowed to cite
Wikipedia. Find that information using the UW Libraries website.
b. Faculty/Grad Student version: Please go to the UW libraries main home page.
You want to mention some statistics about Russia in a paper and want to
quickly get information that can be cited properly. Can you show me where
you’d look for information like that using the UW Libraries website.
13.Reserve a Study Room
a. Return to the main UW libraries home page. You‟re working on a group
project and your group wants to meet in the library to plan the research.
Sign up to use a group study room.
b. Request a Purchase
Faculty/Grad Student version: You’ve found a book you’d like to include in
your next course but the library doesn’t yet own this title. Where would you
go to request that the libraries purchase some copies?
14.Ask Us
a. Return to the main UW libraries home page. You‟re at home working on a
paper that you need to finish tonight, and realize the research is a lot harder
than you expected, so you want to ask a librarian for help. Please do that
now.
b. Faculty/Grad Student version: Return to the main UW libraries home page.
You’re working from home to finish a project that is due tomorrow, and want
to get help from a librarian. Ask a librarian for help with your project using
the website.
15.Library Hours
a. Return to the main UW libraries home page. Tomorrow night you need to stay
in the library to start writing up the outline for your paper. How late is
Suzzallo going to be open tomorrow?
b. Faculty/Grad Student version: Return to the main UW libraries home page.
You need to pick up some materials from Suzzallo on Saturday. How late is
Suzzallo open on the weekends?
Page 36
UW Libraries Website Redesign Usability Evaluation – Autumn 2008
APPENDIX B: ABOUT THIS REPORT
THE STUDY DESCRIBED IN THIS REPORT WAS DESIGNED AND CONDUCTED BY:
Jennifer Ward
Head of Web Services
University of Washington Libraries
Christine Tawatao
Systems Librarian
University of Washington Libraries
Kathryn Whitenton
User Experience Graduate Staff Assistant
University of Washington Libraries
REPORT PREPARED BY:
Kathryn Whitenton
User Experience Graduate Staff Assistant
UW Libraries
For more information about this study, or to request a copy of the unabridged report,
contact libuse@u.washington.edu.
Page 37
Related docs
