CIC Metadata Portal

Compiled by Michelle Dalmau, 11/05/2005, Updated 12/5/2005 With Contributions by Muriel Foulonneau and Tim Cole CIC Metadata Portal Usability Evaluation Plan The Library of the University of Illinois at Urbana-Champaign (UIUC), in collaboration with the other research libraries of the Committee on Institutional Cooperation (CIC), has created and implemented an OAI-PMH metadata harvesting service to aggregate metadata describing information resources held by participating CIC institutions [1]. The UIUC Library has made this metadata aggregation available to end-users (students, faculty, and the general public), both within and outside of the CIC, using appropriate, state-of-the-art search and discovery tools. In collaboration with participating CIC member libraries, the UIUC Library is researching issues relating to consortia metadata aggregation, normalization, and best practice authoring, as well as investigating discovery issues associated with searching, browsing and general navigation that arise in a metadata aggregated environment containing both freely available and restricted license content. By the project’s end, recommendations will be provided to the CIC Library Directors regarding long-term implementation of OAI-PMH and the relationship between OAIPMH and the CIC Virtual Electronic Library (VEL). Background Shreeves & Kirkham and Hagedorn have studied and reported on the major issues related to the representation of aggregated metadata in a coherent service. The CIC metadata portal project team has learned from these studies and has identified solutions such as filtering results by collection in order to avoid long lists of analog-only resources. However, further testing regarding the context provided to resources by collection level descriptions and grouping results by collection is necessary as a way of facilitating readability of heterogeneous aggregated resources. Another area under investigation is the use of thumbnail images and how they assist users in determining relevance. Studies conducted by Dalmau and Dziadosz & Chandrasekar have highlighted the importance of pictorial information in facilitating the choice of relevant resources in a result set. The first study focused on an image database, the second on thumbshots for Websites. They both conclude that the most efficient display for selecting resources from a list of results is a combination of textual and graphic information The CIC metadata does not only include image resources. It also provides thumbnails and thumbshots for collections and for descriptions of Websites, images, articles. The relevance of this approach should be further tested in an aggregation of item level heterogeneous resources. It is notably important to test whether the presence of thumbnails and thumbshots actually modifies the choice of users in investigating the relevance of a resource. Purpose of Study UIUC has designed three interfaces for users to access resources and metadata; a standard search/browse interface, a geographic browse interface and an interface that uses item-level and collection-level descriptive information in combination to facilitate searching for items and browsing of collections [2]. The latter two explore promising methods of access, but are limited -1- Compiled by Michelle Dalmau, 11/05/2005, Updated 12/5/2005 With Contributions by Muriel Foulonneau and Tim Cole in scope and more experimental; for this study we will focus on evaluating the standard, more production-oriented search/browse interface , henceforth called the DLXS website because it utilizes the Digital Library Extension Service (DLXS) middleware to serve the harvested metadata [3]. As noted in the Background section of this plan, various usability studies have been held over the years on similar or evolving OAI provider systems that now culminate as the CIC Metadata Portal. The purpose of this study is to build upon previous studies by formally testing a fully functional web interface in order to understand if the issues uncovered during past iterative testing have indeed been resolved as well as to confirm anecdotal data collected over time regarding use. However, testing described here will be conducted on a version of the interface providing access to all metadata harvested for the project -- i.e., this study will not attempt to address usability issues noted in previous studies relating to aggregation scope and heterogeneity, although in light of earlier studies the tested interface will by default search only resources for which a version can be retrieved in digital form. High-level goals include:  Enhance the design and functionality of the discovery web interface for users in light of the inherent and often difficult to fix problems associated with aggregated metadata Assess potential viability of implemented interface design as basis for a portal that could positively impact research and instruction Understand the potentially diverse audience (librarians, teachers, students, scholars, etc.) who could benefit from using this tool and their specific information needs Understand how users perceive and interact with item-level versus collection-level records o Should item-level simultaneously?   and collection-level descriptions be searched   Specific goals include:  If so, would highlighting collection-level matches alleviate potential confusion caused by decontexualized item-level descriptions? Assess usefulness of having thumbnails included in short-record displays o o o o Does having thumbnails help users more quickly determine results relevance? Will thumbnails automatically inform users that certain resources are available in digital form? Are the “thumbshot” (thumbnail screenshot of websites) thumbnails decipherable? Are items / collections for which thumbnails / thumbshots included selected more often than items / collections not having such added information?  Understand how users interact with potentially voluminous, diverse (in terms of resource format/type, subject, etc.) results set and the website in general -2- Compiled by Michelle Dalmau, 11/05/2005, Updated 12/5/2005 With Contributions by Muriel Foulonneau and Tim Cole o Do users require more sophisticated ways to manipulate the results set (beyond simply sorting) such as filtering (by type, date, digital only, etc.) or keyword refining? Uncover other ways users may be able to drill-down or cluster the result set (i.e. by facets) Assess if current labeling and instructions make sense to a diverse group of representative users Uncover other obstacles to discovery such as metadata issues (name/term variants, lack of authority control, restricted access, etc.) o o o Participants The portal’s contents appeal to a wide variety of people, but we will have to limit the scope to a manageable representative set of participants for this study. Because supporting teaching and research is a primary goal of the portal, we will focus on secondary school teachers, public and academic librarians, faculty members, undergraduates and graduate students. Following is a representative breakdown of participants according to their roles. However, the potential audience of online resources for teaching and learning is wide and the following categories are only provided as an example. 1 9-12 high school teachers 1 Librarians (academic) 2 Faculty members 2 Undergrads 2 Grad students 8 Total Participants Aside from the above-defined roles, another criterion for recruitment is that the participants must use online resources on occasion for teaching and/or learning. Participant recruitment will occur through direct contact either by email or in-person (see Appendix A). Departmental liaisons and listservs will be utilized as vehicles for recruitment. Participants will be compensated for their time with a $15.00 bookstore gift certificate. If participants decide after reviewing the consent form that they do not wish to participate in this study, they will not be awarded the gift certificate. However, participants that continue on, even if they leave part way, will receive compensation. Participants will be able to discontinue their involvement or decline further participation at anytime with no penalty. -3- Compiled by Michelle Dalmau, 11/05/2005, Updated 12/5/2005 With Contributions by Muriel Foulonneau and Tim Cole There are no known risks associated with this study1. Methodology The usability evaluation will consist of two main activities: task completion (participant-led) and retrospective walkthrough (facilitator-led). During task completion, participants will directly interact with the DLXS website as they complete predefined tasks based on actual information needs. After task completion, the facilitator will lead the participants to further explore and expound on various aspects of the DLXS website as they relate to the objectives specified above. Both the task completion and retrospective walkthrough activities, as well as additional questionnaires and debriefing interviews (described below), are designed to collect quantitative, qualitative and satisfaction measures. Quantitative data on interface effectiveness (e.g., the ability to find information) and efficiency (e.g., the ability to locate information within a reasonable amount of time) will be collected. Qualitative data following the think-aloud protocol will also be recorded during task completion and the walkthrough to corroborate efficiency, effectiveness and satisfaction outcomes. Comparative opinions of the DLXS interface evaluated during task completion and during the retrospective walkthrough will also be recorded for analysis. Lastly, participants will also complete a questionnaire and a debriefing interview in order to more fully understand the user experience. A pilot evaluation will be held to make sure the methodology and instruments are sound. The usability study is scheduled to take no more than 1 hour. An evaluation script (see Appendix F) will be used in order to ensure uniform execution and data collection of the activities. Following is a step-by-step summary of the evaluation session (described in more detail below): 1) Review and signing of the consent form, 5 minutes (see Appendix B) 2) Task Completion, 20 minutes (see Appendix D) 3) Retrospective Walkthrough (see Appendix F), 20 minutes 4) Completion of background & satisfaction questionnaire, 10 minutes (see Appendix C) 5) Debriefing/Wrap Up, 5 minutes Evaluation Environment The studies will be conducted either in a fully equipped usability lab that facilitates audio and video screen recording of the sessions or in a portable usability lab setting in which individual computers are set up with software for screen and audio capture. The latter could take place in a computer lab, conference room, offices, etc. The video will only be a computer screen capture. A facilitator and note taker will be present for each study. Notes will be recorded on a standard log form (see Appendix E). Evaluation Measures The following data will be collected: 1 Some of the language in this evaluation plan is included to conform with the IU Human Subjects process. -4- Compiled by Michelle Dalmau, 11/05/2005, Updated 12/5/2005 With Contributions by Muriel Foulonneau and Tim Cole Quantitative Measures Number of errors in task path Number of tasks completed successfully Level of task completion (pass easy, pass hard, fail) Amount of time to complete task Number of clicks per task Likert scale in questionnaires Semantic Differential scale in questionnaires Qualitative Measures Talk-aloud verbal protocol during task completion (coded as positive | negative | both | neutral) Walkthrough comments Behavioral observations Open-ended questions in questionnaires and debriefing interview Data Gathering, Analysis and Reporting All data will be gathered in an anonymous fashion. Participants will only be referenced by a generic identifier (e.g. P1). There will be no way to trace the participant’s name to the assigned generic identifier; consent forms are the only documents that contain personal information (name) and, as such, will be stored in a separate file not linked to data collected for that participant, including the identifier. All materials such as notes, audio and video tapes, etc. will be kept in restricted access in the [indicate premises] offices and will be evaluated only to the researchers of this study. Materials will be destroyed three years after the completion of this study, [indicate date]. The study will entail the following: Completion of forms: Consent Form The signed consent form (see Appendix B) will contain unique identification information such as name, but will be kept in a separate file. There will be no direct mapping from this form to the data collected in the questionnaires and during the card sort and task scenario activities. Task Completion Participants will be presented with 2 tasks (see Appendix D). They will interact with the DLXS website in order to solve the tasks. There are not timed tasks, but if the participants feel they are -5- Compiled by Michelle Dalmau, 11/05/2005, Updated 12/5/2005 With Contributions by Muriel Foulonneau and Tim Cole unable to complete the task, they are free to stop at any time. Conversely, if participants are taking too long during a task, the facilitator should intervene. Participants will be asked to state aloud when they start and stop a task to help with data analysis. Retrospective Walkthrough After task completion, the facilitator will review the DLXS website with the participants in order to further explore issues encountered during task completion as well as explore other areas of interest not necessarily uncovered during task completion. A walkthrough script (see Appendix F) will serve more as a rough guide for the facilitator since many of the issues to be explored are dependent on the task completion outcome. Completion of Background & Satisfaction Questionnaire The background questionnaire (see Appendix C) is used to gather general information about the participant such as academic affiliations, computer experience, etc. Participants will also complete a satisfaction questionnaire (see Appendix C) that is designed to assess key areas of the website they just evaluated: functionality, organization of content, etc. Only generic identifiers such as P1, P2, etc. will be used on this form. Debriefing and Compensation Finally, participants and the researcher are given an opportunity to ask questions for clarification, raise issues, etc. Participants will also be compensated for their time with a $15.00 bookstore gift certificate. Data Analysis Microsoft’s Excel spreadsheet program will be primarily used for analysis. Data can be easily sorted, coded and displayed in various ways including graphical representations. General goals for data analysis have been identified (see list below). Other goals and criteria may surface upon closer inspection of the processed data: Task Completion  Think of a topic you recently researched. Perhaps you needed to learn more about a topic for a research paper or an article or, perhaps, you were looking for particular images to include in a classroom presentation. Please state the topic of interest and conduct a search using the CIC website for that particular topic. o Rational: This is intended as a warm-up task. We would like to diminish any stress that comes with exploring a new, vast resource before the participants perform the pre-defined tasks. o Areas/Aspects Intended to Uncover: Understand actual information needs; learn about resources actually used when initially resolving initial information need; gather first impression of the CIC Portal website: interface layout, functionality, etc.  You are preparing a presentation on the American Civil War (1861-1865). You would like to include historic maps, images that depict general scenes of the war and an excerpt from Abraham Lincoln’s “Gettysburg Address.” -6- Compiled by Michelle Dalmau, 11/05/2005, Updated 12/5/2005 With Contributions by Muriel Foulonneau and Tim Cole o Find at least one Civil War-era historic map.  Rational: This subtask is designed to evaluate how thumbnails (itemlevel access) and thumbshots (collection-level access) affect how the user determines results relevancy. This task is also designed to understand if the user perceives a difference between item- and collection-level access. Areas/Aspects Intended to Uncover: Preference for descriptions with accompanying thumbnails; Learn whether users make distinctions between types of resources (e.g. website versus an image)  o Find at least two images such as portraits of generals, battle scenes, etc. that depict aspects of the American Civil War.  Rational: This subtask is designed for item-level discovery. It also states a certain type of resource, an image. We would like to understand how the user perceives an item-level resource as opposed to a collection-level resource. We would also expect to understand how the user perceives types of resources. Areas/Aspects Intended to Uncover: Search strategy: (will the user filter by type? will s/he want to compose a complex query to tackle all the information need or break up the query into chunks?); Preference for descriptions with accompanying thumbnails? Rational: This subtask is designed for collection-level discovery. It also states a certain type of resource, text. We would like to note if participants realize the variety of content types available so that they can make suggestions as to how to better present and interact with that type of information. Areas/Aspects Intended to Uncover: Search strategy: (will the user filter by type?); Preference for descriptions with accompanying thumbnails?  o Find a website that contains the text for the “Gettysburg Address.”   Retrospective Walkthrough The preliminary categories identified below are by no means comprehensive. They also reflect a limited hierarchy of categorization. Comments regarding results functionality uncovered can be further divided into aspects such as faceted results, sorting, etc.      Scope of the CIC Collection Interface: Organization of Content Interface: Labeling Interface: Description of Resources Interface: Thumbnail Usage -7- Compiled by Michelle Dalmau, 11/05/2005, Updated 12/5/2005 With Contributions by Muriel Foulonneau and Tim Cole    Functionality: Search Functionality: Browse Functionality: Results In the final report, findings will be mostly presented in terms of aggregated results; however, individual responses and quotations may be described. When this is the case, participants will be referenced by their generic identifier, e.g. “P1 said such and such”. Quotations, responses or observations will only be noted for emphasis to support the aggregated data findings. Notes [1] Founded in 1958, the CIC is the academic consortium of twelve research universities including the eleven members of the Big Ten Athletic Conference and the University of Chicago. Together, CIC member universities confer nearly 15% of the Ph.D. degrees awarded in the United States annually, employ more that 33,000 faculty and enroll nearly one-half million students. CIC member institutions include the Universities of Chicago, Illinois, Iowa, Michigan, Minnesota, and Wisconsin-Madison, as well as Indiana, Michigan State, Northwestern, Ohio State, Pennsylvania State, and Purdue Universities. For more information about the CIC, visit their Web site at http://www.cic.uiuc.edu/. [2] To access other interfaces, see . [3] The CIC-OAI version of the DLXS interface was modified in November 2005 to incorporate some of the initial lessons learned from development of and experimentation with the other two interfaces described above. A revised entry page to the interface is also being implemented to better highlight availability of collection browse and provide users with a scope/introduction/aggregation mission statement. References Sarah L. Shreeves and Christine M. Kirkham, Experiences of Educators Using a Portal of Aggregated Metadata in JODI 5-3, 2004, http://jodi.ecs.soton.ac.uk/Articles/v05/i03/Shreeves/ Kat Hagedorn, "OAIster: a no 'dead end' OAI service provider", in Library Hi Tech, the OAIPMH, vol 21 number 2, 2003, p170-181 Michelle Dalmau, Charles W. Cushman Photograph Collection: Report on the Group and Individual Walkthrough, 2003. http://www.letrs.indiana.edu/~mdalmau/cushman/prototype/designDocs/cushWalkFinalReport.p df Susan Dziadosz, Raman Chandrasekar, Do Thumbnail Previews Help Users Make Better Relevance Decisions about Web Search Results? In Proceedings of the 25th annual international ACM SIGIR conference on Research and development in information retrieval, 2002 http://delivery.acm.org/10.1145/570000/564446/p365dziadosz.pdf?key1=564446&key2=3977395111&coll=GUIDE&dl=ACM&CFID=45018362&C FTOKEN=60826084 -8- Compiled by Michelle Dalmau, 11/05/2005, Updated 12/5/2005 With Contributions by Muriel Foulonneau and Tim Cole Appendix A: Email Recruitment Message The following sections present a sample of the messages which will be distributed at Indiana University. Other campuses will use similar material. Introducing CIC Metadata Portal Project The Library of the University of Illinois at Urbana-Champaign (UIUC), in collaboration with the other research libraries of the Committee on Institutional Cooperation (CIC), has created and implemented an OAI-PMH metadata harvesting service known as the CIC Metadata Portal website, which is a service designed to aggregate cultural heritage resources and promote crosscollection discovery of these resources. For more information about the project, visit: http://cicharvest.grainger.uiuc.edu/index.asp. Call for volunteers We are currently investigating how teachers and students browse and search a vast, heterogeneous service such as the CIC Metadata Portal website. The findings from this study will improve the design and functionality of the portal. If you use online resources on occasion for teaching or learning and are interested in participating in the design process, we would benefit greatly from your help. In return, we will compensate your time with a $15.00 gift certificate of the Amazon.com bookstore. What, when and where? We are holding individual usability sessions with a total of 8 participants the week of [DMMYYYY]. The sessions are scheduled to last no more than 1 hour, and will be held in a conference room in the [indicate premises] The usability session entails: 1) completion of consent form; 2) web-based tasks; 3) retrospective walkthrough of the website; 4) completion of a brief questionnaire; and 5) time for final questions and passing of gifts for your participation. We will note your actions, comments, questions raised, etc. during the session. All information recorded is done so anonymously and will be kept confidential. If you are interested in participating, please send me (mdalmau@indiana.edu) available times during the week of [DDMMYYYY]. Please respond as soon as you can with multiple time options so we can schedule the session accordingly. Thanks! Your participation will benefit all users of the CIC Metadata Portal website. Thank you for your time, Michelle Dalmau (mdalmau@indiana.edu, 855-1261) -9- Compiled by Michelle Dalmau, 11/05/2005, Updated 12/5/2005 With Contributions by Muriel Foulonneau and Tim Cole Appendix B: Consent Form Indiana University – Bloomington Informed Consent Statement CIC Metadata Portal –Task-Based Evaluation and Retrospective Walkthrough for Navigating and Exploring Aggregated Resources You are invited to participate in a research study. The purpose of this study is to learn how you search for cultural heritage resources in a vast, heterogonous online resource. Information The Library of the University of Illinois at Urbana-Champaign (UIUC), in collaboration with the other research libraries of the Committee on Institutional Cooperation (CIC), has created and implemented an OAI-PMH metadata harvesting service known as the CIC Metadata Portal website, which is a service designed to aggregate cultural heritage resources and promote crosscollection discovery of these resources. We are in the final design stages and would benefit from your feedback as potential users of this resource. If you chose to participate, you will complete a task-based evaluation and retrospective walkthrough of the website under evaluation that will help us understand how you search for resources to assist you with teaching and/or learning. You will be given three tasks to complete. The retrospective walkthrough is an informal interview that explores the actions and comments you stated while using the CIC Metadata Portal website. You will also be asked to complete a short questionnaire and participate in a debriefing interview. The session will last no more than 1 hour. A facilitator and a note-taker will be present during the session. The sessions will be audio and video recorded. The video will only be a computer screen capture (not your person). While you are completing the activities, you will be encouraged to talk through your thoughts and your actions as you complete the activities. You are also encouraged to ask questions at any time. The questionnaires, audio/video capture and session notes are anonymous; personal identification information, such as your name, will not be referenced on these documents. The questionnaire and sessions notes will only be seen by the researchers and will be used to derive website design recommendations for the CIC Metadata Portal website. Risks - 10 - Compiled by Michelle Dalmau, 11/05/2005, Updated 12/5/2005 With Contributions by Muriel Foulonneau and Tim Cole There are no known risks associated with this kind of study. However, at any time, you are free to take a break during the session. Benefits Your participation will inform the design of the CIC Metadata Portal website, and in turn, will benefit all future users of this cultural heritage service. In addition, the findings from this study may assist other institutions undertaking similar initiatives. Initials: _________ Confidentiality All the data collected will be done so in an anonymous fashion. The questionnaire you will complete will be identified with a generic identifier, e.g. P1, not your name or any other unique identifier such as Social Security Number. The session notes or audio/video recordings will not refer to you by name or by any other identifying characteristic, but they will contain your responses, comments, suggestions and noted observations that arise as you complete the activities. All the data collected will only be reviewed by the researchers of this study. The data, including the questionnaires and session notes, will be kept in restricted access until completion of the CIC Metadata Portal project in December 2006, at which time they will be destroyed. Compensation You will receive a gift certificate to a bookstore worth $15.00 for participating in this study. If you chose to forego participation after reading this consent form, you will not be compensated. Contact If you have questions at any time about this study or the procedures, feel free to ask. You may also contact the researcher, Michelle Dalmau, at the Digital Library Program, Indiana University-Wells Library, 1320 East 10th Street, Room E-170, Bloomington, Indiana 47405, 812-855-1261 and mdalmau@indiana.edu. If you feel you have not been treated according to the descriptions in this form, or your rights as a participant in research have been violated during the course of this project, you may contact the office for the Indiana University Bloomington Human Subjects Committee, Carmichael Center L03, 530 E. Kirkwood Ave., Bloomington, IN 47408, 812/855-3067, by e-mail at iub_hsc@indiana.edu. - 11 - Compiled by Michelle Dalmau, 11/05/2005, Updated 12/5/2005 With Contributions by Muriel Foulonneau and Tim Cole Participation Your participation in this study is voluntary; you may refuse to participate without penalty. If you decide to participate, you may withdraw from the study at any time without penalty and without loss of benefits to which you are otherwise entitled. If you withdraw from the study before data collection is completed your data will be returned to you or destroyed. Consent I have read this form and received a copy of it. I have had all my questions answered to my satisfaction. I agree to take part in this study. Participant’s signature_____________________________________ Date______________ Investigator’s signature____________________________________ Last revised: [date] Date______________ - 12 - Compiled by Michelle Dalmau, 11/05/2005, Updated 12/5/2005 With Contributions by Muriel Foulonneau and Tim Cole Appendix C: Background & Satisfaction Questionnaire Participant Background & Satisfaction Questionnaire CIC Metadata Portal Website 1. Status (circle all that apply): Faculty Undergraduate Student Librarian Non-IU Instructor/Teacher Graduate Student a. If Non-IU, please specify: _______________________________________________ b. Area(s) of study, expertise or professional practice: ______________________________________________________________________ 2. What type of computer do you use? (circle all that apply) a. at home: b. at work: PC PC MAC MAC 3. About how many hours a week do you spend online (WWW, email, etc)? < 5 hrs 6-10 hrs 11-15 hrs 16-20 hrs 21-25 hrs 26-30 hrs > 30 hrs 4. Which browser(s) do you most commonly use? (e.g. IE 6.0, NS 4.7, Firefox, Safari) ____________________________________________________________________________ 5. Please indicate with an “x” how frequently you use online resources for teaching and/or learning: Never __ __ __ __ __ Always a. List a few of the online resources you most commonly use: ________________________________________________________________________ ________________________________________________________________________ 6. Describe why and how you typically use online resources in your teaching or research. ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ ______________________________________________________________________________ - 13 - Compiled by Michelle Dalmau, 11/05/2005, Updated 12/5/2005 With Contributions by Muriel Foulonneau and Tim Cole 7. Please tell us what you think of the major areas of the CIC Metadata Portal website. Feel free to visit the website to refresh your memory. Navigation Organization of Content Browsing Searching Results Other 8. Your overall reaction to the websites? Feel free to comment on any of the statements or questions below. Unattractive __ __ __ __ __ Attractive Frustrating __ __ __ __ __ Satisfying Easy to Use __ __ __ __ __ Hard to Use Likable __ __ __ __ __ Unlikable Confusing __ __ __ __ __ Clear Effective __ __ __ __ __ Ineffective Pleasant __ __ __ __ __ Unpleasant Hard to Use__ __ __ __ __ Easy to Use Organized __ __ __ __ __ Disorganized Learn Easy __ __ __ __ __ Learn Hard Satisfied __ __ __ __ __ Dissatisfied Productive __ __ __ __ __ Unproductive Comments: ___________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ 9. Labels on the screen are? Feel free to comment on any of the statements or questions below. - 14 - Compiled by Michelle Dalmau, 11/05/2005, Updated 12/5/2005 With Contributions by Muriel Foulonneau and Tim Cole Inconsistent __ __ __ __ __ Consistent Confusing __ __ __ __ __ Clear Comments: ___________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ 10. Information displayed on the screen is? Feel free to comment on any of the statements or questions below. Inconsistent __ __ __ __ __ Consistent Confusing __ __ __ __ __ Clear Comments: ___________________________________________________________________________ _____________________________________________________________________________________ 11. Navigation is? Feel free to comment on any of the statements or questions below. Inconsistent __ __ __ __ __ Consistent Confusing __ __ __ __ __ Clear Helpful __ __ __ __ __ Unhelpful Easy to Use __ __ __ __ __ Hard to Use Intuitive __ __ __ __ __ Unintuitive Comments: ___________________________________________________________________________ _____________________________________________________________________________________ 12. Website capabilities are? Feel free to comment on any of the statements or questions below. Hard Search __ __ __ __ __ Easy Search Hard Browse __ __ __ __ __ Easy Browse Confusing Search __ ___ ___ __ __ Clear Search Confusing Browse __ __ __ __ __Clear Browse Sort Options Good __ __ __ __ __ Sort Options Bad Grouping Results Helpful __ __ __ __ __ Grouping Results Unhelpful Comments: ___________________________________________________________________________ _____________________________________________________________________________________ - 15 - Compiled by Michelle Dalmau, 11/05/2005, Updated 12/5/2005 With Contributions by Muriel Foulonneau and Tim Cole 13. What aspects of the site did you find particularly useful? _____________________________________________________________________________________ _____________________________________________________________________________________ 14. What aspects of the site did you find particularly problematic? _____________________________________________________________________________________ _____________________________________________________________________________________ Thank you for completing this questionnaire. - 16 - Compiled by Michelle Dalmau, 11/05/2005, Updated 12/5/2005 With Contributions by Muriel Foulonneau and Tim Cole Appendix D: Tasks 1. Think of a topic you recently researched. Perhaps you needed to learn more about a topic for a research paper or an article or, perhaps, you were looking for particular images to include in a classroom presentation. Please state the topic of interest and conduct a search using the CIC website for that particular topic. 2. You are preparing a presentation on the American Civil War (1861-1865). You would like to include historic maps, images that depict general scenes of the war and an excerpt from Abraham Lincoln’s “Gettysburg Address.” a. Find at least one Civil War-era historic map. b. Find at least two images such as portraits of generals, battle scenes, etc. that depict aspects of the American Civil War. c. Find a website that contains the text for the “Gettysburg Address.” - 17 - Compiled by Michelle Dalmau, 11/05/2005, Updated 12/5/2005 With Contributions by Muriel Foulonneau and Tim Cole Appendix E : Observation Log Evaluator :________________ Actions Taken Participant # :_______ Obersvations : Comments, Questions, Gestures - 18 - Compiled by Michelle Dalmau, 11/05/2005, Updated 12/5/2005 With Contributions by Muriel Foulonneau and Tim Cole - 19 - Compiled by Michelle Dalmau, 11/05/2005, Updated 12/5/2005 With Contributions by Muriel Foulonneau and Tim Cole Appendix F : Script for Tasks and Retrospective Walkthrough [Introductions : Orientation (bathrooms, cafeteria, etc.)] [Review of Consent Form : Questions ?] [Overview of CIC Metadata Portal Project] The Library of the University of Illinois at Urbana-Champaign (UIUC), in collaboration with the other research libraries of the Committee on Institutional Cooperation (CIC), has created and implemented an OAI-PMH metadata harvesting service known as the CIC Metadata Portal website, which is a service designed to aggregate cultural heritage resources and promote crosscollection discovery of these resources. [Overview of Activities] You will help us finalize the design of the CIC Metadata Portal website by evaluating the website and providing feedback. You will complete two activities: a task-based evaluation and a retrospective walkthrough. You will also complete a short questionnaire at the end. While completing the tasks, I will encourage you to talk-aloud. We are interested to know what you are thinking as you work through the tasks. You can ask me questions at any time. [Note taker’s name] will be taking notes. You are free to look at [his/her] notes after each activity. [Administer Tasks / Debrief] You will have a total of three tasks to complete. You will be given each task one at a time. Please read the task and let me know when you are ready to start. These tasks are not timed. You can move on to the next task at any time. I may also encourage you to move on if you encounter problems with the website. This should take no more than 20 minutes. [Administer Retrospective Walkthrough / Debrief] Now we will conduct a retrospective walkthrough of the website. This means that we will review areas of the website you visited, how you interacted with the website during task completion, etc. We will also explore other aspects of the website you may not have - 20 - Compiled by Michelle Dalmau, 11/05/2005, Updated 12/5/2005 With Contributions by Muriel Foulonneau and Tim Cole encountered. We would like your general impression, positive or negative, of the website. This should take no more than 20 minutes. Search     What are you general impressions of this page? Did you encounter any problems using the search page? What is missing on this page? Do you expect to do certain things that you are unable to do? [Address any issues that arose with this page during task completion] Browse & Browse Results   What did you expect to happen when you clicked on the browse link? What are you general impressions of the browse results page? o What do you think of the labels used? o What do you think of items that are linked?    What do you expect the linked title to do? the linked homepage? What are your impressions about the way the browse results page is organized? o Can you suggest other ways to improve your interaction with the results page to help you more quickly find what you are looking for?  What do you think about browsing by collection? Let’s try browsing by type. What do you think of that? o What do you think “browsing by collection” means? o What do you think “browsing by type” means?     What do you think of the thumbnails? o How do they or don’t they help you make sense of the results? Is there enough information on this page to help you select a resource of interest? If not, what’s missing? What do you think about the left-hand side of the page? How do those links help you navigate the website? [Address any issues that arose with this page during task completion] Results - 21 - Compiled by Michelle Dalmau, 11/05/2005, Updated 12/5/2005 With Contributions by Muriel Foulonneau and Tim Cole   Let’s try searching on a variation of task 2. Please type “civil war” and “united states.” What are your general impressions of the results page? [Scroll down] [Go to next page and scroll down] o What do you think of the labels used? o What do you think of items that are linked?   What do you expect the linked title to do? the linked collection name? o What do you think of the sort option? Of the “expand search” option? Of the “refine search” option?  What are your impressions about the way the results page is organized? o Can you suggest other ways to improve your interaction with the results page to help you more quickly find what you are looking for?     What do you think of the thumbnails? o How do they or don’t they help you make sense of the results? Is there enough information on this page to help you select a resource of interest? If not, what’s missing? What do you think about the left-hand side of the page? How do those links help you navigate the website? [Scroll down and click on U of Mich, Art, Architecture Library then click on the first record] o What did you expect when we clicked on the title link of the first record?  Do you find this interim page helpful or not? o [Click on image link] What did you expect to find when we clicked on this image link?   How do you feel about the restricted access message? [Address any issues that arose with this page during task completion] [Wrap up: Final Questions/Comments, Gift Disbursement] - 22 -