INLS 257, Spring 2004
28, USABILITY TEST EXAMPLE: MENU NAVIGATION
Zaphiris, P., Shneiderman, B., & Norman, K. L. (2002). Expandable indexes vs. sequential menus for searching hierarchies on the World Wide Web. Behaviour & Information Technology, 21(3), 201-207. *** PP slide 1. Project Description (a) Purpose, hypotheses or research questions To investigate the navigation of two different menu types (hierarchical expanding and sequential) at three levels of depth *** PP slide What do we already know about this design problem? (outside AA-IRB forms) Many studies of depth vs. breadth; cited and discussed Pretty good example of a lit review Criteria for success *** PP slide Speed/response time Retrieval/search efficiency Subjective ratings/preference (b) Procedures Research design *** PP slide 3x2 factorial design Between-subjects on depth Within-subjects on type of menu Introduce the participant to the purpose of the study Get informed consent Participant assigned to one of the three levels of menus Complete 10 assigned tasks as quickly as possible *** PP slide All tasks: locate the number of employees in a specific industry Based on 1996 Annual Survey of Manufacturers’ Statistics for Industry Groups and Industries Half of the tasks with expandable menu and half with sequential menu Response time and search efficiency were measured from transaction logs (time stamps and mouse clicks) Subjective ratings collected after completion of all 10 tasks
28, Example usability test, Zaphiris et al., 2002, page 2 INLS 257, Spring 2004
2. Participants (a) Age, sex, and approximate number Mean age was 27 (variability not reported) 14 males, 7 females 21 total (b) Inclusion/exclusion criteria, if any All had at least an undergraduate degree in business and management (c) Method of recruiting Not specified (d) Inducement of participation None described 3. Are participants at risk? *** PP slide No risk in this study 4. Describe steps to minimize risk Not applicable 5. Are illegal activities involved? No 6. Is deception involved? No; “real” purpose of study explained to participants at beginning 7. What are the anticipated benefits to participants and/or society? Better design of Web menus 8. How will prior consent be obtained? *** PP slide After study purposes explained Prior to beginning interaction with system Written informed consent 9. Describe security procedures for privacy and confidentiality. *** PP slide During data collection Only necessary if data is something sensitive Protecting the data Keep physical media in locked cabinets If on computer, keep access to it protected physically or via password
28, Example usability test, Zaphiris et al., 2002, page 3 INLS 257, Spring 2004
Results *** PP slide Data analysis primarily analysis of variance Examined each dependent variable separately Response time: Figure 3, Table 1 Sequential menus were faster Depth made more of a difference for expandable menus Search efficiency: Figure 4, Table 2 No main effects Depth had an effect on expandable menus Example of interaction effects, Figure 4 *** PP slide Subjective ratings/preferences: Table 3 No statistically-significant differences Statistical significance vs. effect size Response time Sequential menus: 35-41 seconds per item Expandable menus: 40-66 seconds per item Search efficiency Sequential menus: 0.57-0.97 off-path mouse clicks Expandable menus: 0.09-2.34 off-path mouse clicks Subjective ratings/preferences No differences to interpret Implications Conclusions for design: Don’t use expandable menus If you do, only use them for shallow menu structures