An Introduction to Usability Testing by gregoria

VIEWS: 0 PAGES: 102

									An Introduction to Usability Testing
Bill Killam, MA CHFP
Adjunct Professor University of Maryland bkillam@user-centereddesign.com

User-Centered Design  www.user-centereddesign.com

Definitions
 “Usability testing” is the common name for user-based system evaluation  Popularized in the media by Jakob Neilson and usually thought of as related to web site design in the 1990’s  Usability testing is one of the activities of Human Factors Engineering

2

User-Centered Design  www.user-centereddesign.com

Some Historical Examples
 1911 – Taylor study of “which is the best way to do a job?” and “what should constitute a day’s work” to determine time standards for basic tasks

 1911 - Frank (an Industrial Engineer) and Lillian (a Psychologist) Gobreth, studied the motions involved in bricklaying – reduced motions from 18 to 5, the development of the “therblig” unit of motion  Late 1940, psychologist at Write Patterson Air Force Base studies crashes to determine the cause, it was not what was expected  A study on the effect of a redundant, high-centered taillight on rear end car crashes

3

User-Centered Design  www.user-centereddesign.com

What is the System
 Sid Smith’s “user-system interface” (compared to user-computer interface)  Systems are made up of users performing some activity within a context  Can’t redesign users but we can design equipment, so our goal as designer is to “design equipment to optimize system performance”

4

User-Centered Design  www.user-centereddesign.com

What does “usability” mean?
 Accepted Definition
– The ability of a specific group of users to perform a specific set of activities within a specific environment – The ability of a specific group of users to perform a specific set of activities within a specific environment with effectiveness, efficiency,

 ISO 9241 Definition

and satisfaction (emphasis added)
Satisfaction Efficiency Learnability Low Errors Memorability

 Neilson
– – – – –

5

User-Centered Design  www.user-centereddesign.com

Contributors to Usability
 Functional Suitability – does the product contain the functionality required by the user?  Ease-of-learning – can the user figure out how to exercise the functionality provided  Ease-of-use - can the user exercise the functionality accurately and efficiently (includes accessibility issues)  Ease-of-recall – can the knowledge of operation be easily maintained over time?  Subjective Preference

6

User-Centered Design  www.user-centereddesign.com

There are a lot of factors…
…our brain’s work against us…
User-Centered Design  www.user-centereddesign.com

+

•

8

User-Centered Design  www.user-centereddesign.com

9

User-Centered Design  www.user-centereddesign.com

10

User-Centered Design  www.user-centereddesign.com

11

User-Centered Design  www.user-centereddesign.com

Red Green Blue Orange Yellow Black

12

User-Centered Design  www.user-centereddesign.com

Stroop Stroop Stroop Stroop Stroop Stroop

Orange Yellow Green

13

User-Centered Design  www.user-centereddesign.com

Black Blue Red

14

User-Centered Design  www.user-centereddesign.com

15

User-Centered Design  www.user-centereddesign.com

…we have trouble with patterned data…
User-Centered Design  www.user-centereddesign.com

17

User-Centered Design  www.user-centereddesign.com

18

User-Centered Design  www.user-centereddesign.com

19

User-Centered Design  www.user-centereddesign.com

20

User-Centered Design  www.user-centereddesign.com

21

User-Centered Design  www.user-centereddesign.com

22

User-Centered Design  www.user-centereddesign.com

23

User-Centered Design  www.user-centereddesign.com

24

User-Centered Design  www.user-centereddesign.com

…we can be fooled…
User-Centered Design  www.user-centereddesign.com

26

User-Centered Design  www.user-centereddesign.com

27

User-Centered Design  www.user-centereddesign.com

28

User-Centered Design  www.user-centereddesign.com

29

User-Centered Design  www.user-centereddesign.com

30

User-Centered Design  www.user-centereddesign.com

31

User-Centered Design  www.user-centereddesign.com

32

User-Centered Design  www.user-centereddesign.com

33

User-Centered Design  www.user-centereddesign.com

34

User-Centered Design  www.user-centereddesign.com

35

User-Centered Design  www.user-centereddesign.com

…we make perceptual assumptions…
User-Centered Design  www.user-centereddesign.com

37

User-Centered Design  www.user-centereddesign.com

38

User-Centered Design  www.user-centereddesign.com

39

User-Centered Design  www.user-centereddesign.com

40

User-Centered Design  www.user-centereddesign.com

Jack and Jill went went up the Hill to fetch a a pail of milk

41

User-Centered Design  www.user-centereddesign.com

FINISHED FILES ARE THE RESULT OF YEARS OF SCIENTIFIC STUDY COMBINED WITH THE EXPERIENCE OF MANY YEARS

42

User-Centered Design  www.user-centereddesign.com

…our perceptual abilities are limited in the presence of noise…
User-Centered Design  www.user-centereddesign.com

THE QUICK BROWN FOX JUMPED OVER THE LAZY DOG’S BACK.
44
User-Centered Design  www.user-centereddesign.com

The quick brown fox jumped over the lazy dog’s back.

45

User-Centered Design  www.user-centereddesign.com

THE QUICK BROWN FOX JUMPED OVER THE LAZY DOG’S BACK.
46
User-Centered Design  www.user-centereddesign.com

The quick brown fox jumped over the lazy dog’s back.

47

User-Centered Design  www.user-centereddesign.com

…our cognitive abilities are limited…

User-Centered Design  www.user-centereddesign.com

The Game of 15’s
 Let’s play the game of “15.” The pieces of the game are the numbers 1, 2, 3, 4, 5, 6, 7, 8, and 9. Each player takes a digit in turn. Once a digit is taken, the other player cannot use it. The first player to get three digits that sum to 15 wins.  Here’s a sample game: Player A takes 8. Player B takes 2. Then A takes 4, and B takes 3. A takes 5. What digit should B take?

49

User-Centered Design  www.user-centereddesign.com

X

O X

X

O

50

User-Centered Design  www.user-centereddesign.com

4 9 2

3 5 7

8 1 6

51

User-Centered Design  www.user-centereddesign.com

…our memory affects our abilities…

User-Centered Design  www.user-centereddesign.com

…our psychology affects our abilities…

User-Centered Design  www.user-centereddesign.com

…and we need protection from ourselves…
User-Centered Design  www.user-centereddesign.com

55

User-Centered Design  www.user-centereddesign.com

1st Dimension of Usability
 Functional “Visibility” through an obvious “visible” structure and adequate feedback
– (1) Affordances

• • • • • • • • •

Perceived and actual properties of things Chairs are for sitting on Doors are for opening Flat surfaces are for placing things on Buttons are for pressing Plates are for pushing Handles are for pushing Knobs are for turning Slots are for inserting things

56

User-Centered Design  www.user-centereddesign.com

1st Dimension of Usability (concluded)
– (2) Constraints • Obvious intended use or limitations (e.g., intended use:

– (3) Mapping • Relationship between parts (e.g., control movement and effects in the real world) • Natural mapping – conforms to cultural norms or physical analogies

mouse is for grabbing with the hand, trackball is for rolling with the thumb; limitation: buttons are presses with the finger, pedals are stepped on)

 Adequate Feedback – required to support, confirm, and reinforce the visible structures

57

User-Centered Design  www.user-centereddesign.com

2nd Dimension of Usability
 Good (conceptual) model to predict the effects of our actions
– what actions are intended – what the consequences of our actions will be – what to do if something goes wrong

 Examples
– Real World Metaphors - Trash Can icon, Shopping Carts – Difficult Models – multiple drive types, phone purchase, Rubber Stamp

58

User-Centered Design  www.user-centereddesign.com

Issues with Conceptual Modal
 Sowa

– The development of percepts into a mosaic – Incomplete – Inconsistent – Self Contradictory – Freezer Controls – Ollie North

 Characteristics of Bad Conceptual Model

 Examples

59

User-Centered Design  www.user-centereddesign.com

3rd Principle of Usability
 Design for the Intended user (and not for yourself  Match the Representation to the Task  Example – the Game of 15s in the real world

60

User-Centered Design  www.user-centereddesign.com

1131

SAN

0820+1

LGW

AA

2734

FCYBM

D10

1

AA

2734

CHG PLANE AT DFW

X12

1805

SAN

1425+1

LGW

BA

284

FJMSB

D10

1

2100

SAN

2030+1

LHR

TW

702

FCYBQ

*

2

TW

702

EQUIPMENT 767 LAX L-10

61

User-Centered Design  www.user-centereddesign.com

AA 2734

Gatwick

Gatwick BA 284

TWA 702

Heathrow

Local 6:00 AM (London)

Noon

6:00 PM Midnight 6:00 AM Noon 6:00 PM Midnight (1:00 PM) (7:00 PM) (1:00 AM) (7:00 AM)

62

User-Centered Design  www.user-centereddesign.com

4th Principle of Usability
 Design for Errors and Slips (or “Don’t Blame the User”)

 Desired Characteristics
– immediately detectable feedback – impact should be minimal – results should be reversible

 Levels of Error Design
– – – – Eliminate Protect Warn Train

63

User-Centered Design  www.user-centereddesign.com

How does Usability Testing differ from other forms of “UserBased” activities?
User-Centered Design  www.user-centereddesign.com

Were Not Doing Behavioral Analysis
 Behavior is a specific action or set of actions  Performance is acting towards a goal or objective with a specific measure of success  Behavior and performance are not the same and forcing a behavior rarely ensure proper or even adequate performance

65

User-Centered Design  www.user-centereddesign.com

Market Research
 Market Research (Qualitative & Quantitative) is directed primarily at understanding the actual or potential user population for a product of service
– Size of markets – Reasons for purchasing

66

User-Centered Design  www.user-centereddesign.com

Basic or Even Applied Research
 Large scale effort that requires knowledge of the degrees of freedom between the target population and the sample population and requires statistical analysis to determine the level of significance of results

67

User-Centered Design  www.user-centereddesign.com

User Acceptance Testing
 A script driven process to ensure the functional specifications have been met
– i.e., the functionality exists (regardless of if it is “usable”) – It occurs after development, just before a product is shipped

68

User-Centered Design  www.user-centereddesign.com

When and What

User-Centered Design  www.user-centereddesign.com

When is Usability Testing Conducted?

 Once  A few times during development  Throughout the Development Cycle  Longitudinally

70

User-Centered Design  www.user-centereddesign.com

What do you test?
 Isolating the Variable
– – – – – – Conceptual Designs Architectures Labels Wireframe Design Visual Design Mock-up versus Prototypes • The much maligned paper prototype • Non-operational interfaces and the Wizard of Oz technique

 Comparative Evaluations

71

User-Centered Design  www.user-centereddesign.com

The Basics

User-Centered Design  www.user-centereddesign.com

Usability Testing
 Human Factors Engineering (including usability testing) occurs as part of the design process (typically referred to as a “user-centered design process”) to allow designers to make design decisions based on collected data  Usability testing may be applied as part of an IV&V effort, but a criteria must be established prior to testing

73

User-Centered Design  www.user-centereddesign.com

Types of Usability Testing Used During Design
 Non-User Based
– – – – – – – – –

Expert Review Compliance Reviews Heuristic Evaluations Cognitive Walkthroughs User Surveys Ethnographic Observation Performance-based Think Aloud Co-Discover

 User-based

74

User-Centered Design  www.user-centereddesign.com

Non-User based Testing: Expert Review
 One or two usability experts review a product, application, etc.  Free format review  Subjective but based on sound usability principles  Lowest cost usability testing  Highly dependent on the qualifications of the reviewer(s)

75

User-Centered Design  www.user-centereddesign.com

Non-User based Testing: Compliance Testing
 Style Guide-based Testing
– Interpretation – Checklists – Scope Limitations – Interpretation

 Interface Specification Testing

 Standards-based Testing

– ADA Testing – Ex. Public Law 508 Testing – Ex. DOD DII HCI Interface Specification Testing

76

User-Centered Design  www.user-centereddesign.com

Non-User based Testing: Heuristic Evaluation
 Structured review based on known rules of thumb  Nielson’s 10 Most Common Mistakes Made by Web Developers (three versions)  Shneiderman’s 8 Golden Rules  Norman’s 4 Principles of Usability

77

User-Centered Design  www.user-centereddesign.com

Non-User based Testing: Cognitive Walkthrough
 Team Approach  Issues related to cognition (understanding) more than presentation or sequence control  Subjective but based on sound psychological issues  Also lowest cost usability testing  Highly dependent on the qualifications of the reviewer(s)

78

User-Centered Design  www.user-centereddesign.com

Sidebar: Intrusive versus Non-intrusive
 Projected Responding  The Hiesenberg Uncertainty Principle of Usability  Non-intrusive Testing
– Field Data Collection – Ethnographic Observation (aka Contextual Inquiry)

 Intrusive testing
– The use of controlled environments (repeatability) – Isolation of specific variables

79

User-Centered Design  www.user-centereddesign.com

User-Based Testing: User Surveys
 Pro: Inexpensive and can be conducted remotely

 Pro: Can provide trend data
 Con: Relies on user self reported data  Con: “The vocal minority”

80

User-Centered Design  www.user-centereddesign.com

User-Based Testing: Ethnographic Observation

 Pro: The most “real”

 Con: Analysis Intensive
 Con: No interaction (follow-up)  Con: Ethically challenging

81

User-Centered Design  www.user-centereddesign.com

User-Based Testing: Think Aloud Protocols
 Probably the most common form of usability testing in use today  Pro: Designed to capture participant’s understanding  Con: A disruptive test, cannot be used to evaluate performance  Con: Biased against performance measures

82

User-Centered Design  www.user-centereddesign.com

User-Based Testing: Co-Discovery Protocols
 Variation on Think Out Loud protocol  Multiple participant perspective  Pro: More natural interaction than Think Out Loud  Pro: More fun, more revealing  Con: Potentially more difficult – participant selection/matching issues

83

User-Centered Design  www.user-centereddesign.com

User-Based Testing: Performance-based
 Semi-Intrusive  Pro: Provides an “objective” measure  Pro: Good for comparative evaluations  Con: Not a complete picture of usability, possibly misleading  Critical Incidence Analysis
– Pro: Provides a combination of performance and think aloud in a single session – Con: Risks confabulation

84

User-Centered Design  www.user-centereddesign.com

User-Based Testing: Mental Workload
 Formats…
– – – – – Physiological measures Mental fatigue (performance measures) Blink rate Subjective assessment (Cooper-Harper) Secondary Task Problem

 Pro: Non-intrusive  Con: Difficult to administer and interpret

85

User-Centered Design  www.user-centereddesign.com

User-Based Evaluation: Subjective Measures

 Self-reported ease-of-use measures (summative evaluations)
– SUS – QUIS – SUMI

 Aesthetic value
 User preferences

86

User-Centered Design  www.user-centereddesign.com

How do you Design and Conduct a User-based Test
User-Centered Design  www.user-centereddesign.com

Test Set-up
 Select a Protocol

 Define Your Variables
– Dependent and Independent Variables – Confounding Variables – Operationalize Your Variables

 Formats
– Between Subject Designs – Within Subject Designs

88

User-Centered Design  www.user-centereddesign.com

Participant Issues
 Addressing All Possible User-types (good luck)  How many?
– Relationship to statistical significance – “Pilot Study” format – Discount Usability – who’s rule? – Screeners – Getting Subjects

 Selecting subjects
• Convenience Sampling • Recruiting

 Participant stipends  Over recruiting

 Scheduling
89
User-Centered Design  www.user-centereddesign.com

Defining Task Scenarios
 Areas of concern redesign, or client interest

 Short, unambiguous tasks to be performed
 Wording is critical
– In the user’s own terms – Does not contain “seeds” to the correct solution

 Enough to form a complete test but able to stay within the time limit

90

User-Centered Design  www.user-centereddesign.com

Preparing Test Materials
 Consent form

 Video release form
 Receipt and confidentiality agreement  Demographic form  Introductory comments  Participant task descriptions  Questionnaires, SUS, etc.  Note Takers Forms  Facilitator’s Guide
91
User-Centered Design  www.user-centereddesign.com

Piloting the Design
 Dry running the entire experiment

 Getting subjects
– Convenience sampling

 Collect data

92

User-Centered Design  www.user-centereddesign.com

Collecting Data
 “Superman” versus Facilitator/Observer  Collecting interaction data

 Collecting observed data
– Behavior – Reactions

 Collecting participant comments

 Collecting subjective data
– Pre-test data – Post scenario data – Post test data

93

User-Centered Design  www.user-centereddesign.com

Analyzing Data
 Descriptive versus Predictive Statistics

 Statistical Analysis
– T-test – F-test

 Statistical Significance vs. The Principle of “Inter-ocular Drama”

94

User-Centered Design  www.user-centereddesign.com

Reporting the Results
 Briefing Results  Written Reports  Highlights Tapes
– The “a picture is worth a thousand words” principle

 NIST CIF

95

User-Centered Design  www.user-centereddesign.com

Types of Usability Testing Used During IV&V
 Non-User Based
– – – – – – – – –

Expert Review Compliance Reviews Heuristic Evaluations Cognitive Walkthroughs User Surveys Ethnographic Observation Performance-based Think Aloud Co-Discover

 User-based

96

User-Centered Design  www.user-centereddesign.com

Wrap-up

User-Centered Design  www.user-centereddesign.com

Misc. Things
 When is Usability Testing Conducted? Once
– A few times during development – Throughout the Development Cycle – Longitudinally

 Mock-up versus Prototypes versus Actual Systems/Applications
– The much maligned paper prototype – Non-operational interfaces and the Wizard of Oz technique

 Comparative Evaluations

98

User-Centered Design  www.user-centereddesign.com

Testing Special Situations
 PDA’s, Cell Phones and other Handheld devices  Telephone and IVR interfaces  Remote Usability Testing

99

User-Centered Design  www.user-centereddesign.com

Testing Special Populations
 Kids
– – – – – Consent Incentives Parental Issues Process Management Administering Questionnaires and other data collection issues Special Interface devices Interaction protocols Communications Administering Questionnaires and other data collection issues

 Users with Disabilities
– – – –

100

User-Centered Design  www.user-centereddesign.com

References

User-Centered Design  www.user-centereddesign.com

References
 Handbook of Human Factors Test and Evaluation, O’Brian & Charlton

 A Practical Guide to Usability Testing, Dumas & Redish
 Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests, Jeffery Rubin  Cost-Justifying Usability, Randolph Bias & Deborah Mayhew

102

User-Centered Design  www.user-centereddesign.com


								
To top