21st Century User Testing: Test It, Cheap Fast and Often

							                   21st Century User Testing: Test It Cheap, Fast and Often




    21st Century User Testing: Test It
    Cheap, Fast and Often




1                                                             © Sapient Corporation 2011
                                                                      21st Century User Testing: Test It Cheap, Fast and Often




    21st Century User Testing: Test It
    Cheap, Fast and Often

    Four decades of usability testing has done little to make our            believing that we can somehow create reports that are so
    world any more usable. This is a shame because usability                 effective that stakeholders find them irresistible. But that’s
    testing is such a unique and powerful development tool.                  like believing there’s an umbrella so effective that it can
    By watching people work through specific tasks using the                 stop rain from falling.
    interfaces, applications and services we build, we avoid our
    own biased views of what people want and need. It provides               We kill ourselves to catalogue every possible usability issue,
    a much greater chance for us to meet users’ requirements                 but we need to step back and ask, how many issues from
    and align with their expectations.                                       such an exhaustive list actually get resolved in a typical
                                                                             project? 20 percent? If we got to even half of the issues
    You would expect the use of such an impressive tool for 40               raised by testing, it would be a miracle.
    years to have a positive influence on the overall usability of
    all solutions. So why hasn’t this been the case?                         For years, we had long debates about the appropriate
                                                                             number of usability test participants until finally the
    For one thing, clients may listen intently when we present our           research determined that 85 percent of problems surfaced
    findings, but the data doesn’t seem to stick. In organizations           with five test participants, and we all nodded our heads and
    with weak project management, the data doesn’t stick                     felt secure in our methods. But what practical value have
    because no one is directly responsible for acting on                     we seen in 85 percent discovery when we have only been
    usability testing results. In organizations with strong project          able to address a handful of issues?
    management, the data doesn’t stick because—whether or
    not it fits the project manager’s job description—everyone               Even with all of its flawed elements, traditional practices
    assumes that the project manager will integrate research                 have given us glimpses of usability testing’s true potential.
    results into projects. This means that no one other than the             User Interface Engineering (UIE), a leading user research
    project manager takes personal responsibility.                           firm, tested an e-commerce Web site for a major corporate
                                                                             client where first-time buyers encountered a screen similar
    Another influence on usability testing’s lack of long-term               to the one in Figure 1.
    impact is the way we’ve traditionally reported results. We
    craft the most comprehensive analysis and communication
    of our research possible, but despite our efforts people
    rarely read the report. Many of us have taken this personally,
    seeing it as a reflection on the quality of our work and




2                                                                                                                       © Sapient Corporation 2011
                                                                    21st Century User Testing: Test It Cheap, Fast and Often




                                                                           There is no magic that will make traditional usability testing
                                                                           data suddenly stick within an organization, no secret formula
                                                                           for creating reports that stakeholders will actually read, and
                                                                           no new process that will allow projects to address a majority
                                                                           of issues raised. The only real answer is to radically change
                                                                           how we do usability testing, with the most radical part being
                                                                           that we actually need to be doing less.


                                                                           THE WAY WE’VE ALWAYS DONE USABILITY
    Figure 1. The Login Screen with a Register Button
                                                                           TESTING
    Participants resented being forced to register before
    making a purchase. As one person said, “I’m not here to                This is how it typically works. You pay a recruiter to find
    enter into a relationship, I just want to buy something.” The          people similar to your users. You rent a facility and invite
    client’s intent had been to make shopping easier, but their            key stakeholders to observe the test. Your test participants
    solution actually made it much more difficult. Based on the            work their way through a series of tasks using a fully- or
    usability testing results, the client changed the “Register”           semi-functional version of the Web site or application your
    button to a “Continue” button (as shown in Figure 2). This             team is building.
    made it more obvious that registering was optional.
                                                                           You review the video of each session again and again
                                                                           until you’ve gleaned every possible data point about every
                                                                           possible usability issue. You combine, compare, and classify
                                                                           all of the data and; issues are prioritized and placed into a
                                                                           comprehensive catalog of concerns.

                                                                           You take screenshots from the interface and snapshots
                                                                           of the testing and combine them with your insightful,
                                                                           exhaustive analysis. Then you deliver it all to the client in
                                                                           one giant, glorious report.
    Figure 2. The Login Screen with a Continue Button
    Starting the day they changed the button, the number                   In a meeting you present your data and watch as your client
    of customers purchasing jumped 45 percent. In the first                scrawls notes on the report cover. After skimming the
    month, the client made $15 million more than it had in the             executive summary, the client tosses your shiny, full-color,
    month before the button was changed. In the first year, it             indexed report on a shelf with a stack of others, and she
    made $300 million more in sales than it had the year before.           buries the digital version deep in her hard drive. Your keen
                                                                           insights and analysis eventually blend in with the existing
    Despite hundreds of thousands of usability tests and                   white noise of her organization’s developmental mythology.
    projects as successful as the $300,000 button example,
    the quality of usability across all products and interfaces            If you’ve presented well and stayed within your budget—
    hasn’t improved. It certainly wasn’t the result of a lack of           usually between $20,000 and $40,000, depending on the
    effort. We strove to find the right number of participants to          number of types of users and your team’s travel costs—you
    surface the greatest number of issues. We carefully crafted            get the chance to do it all over again for a later phase or for
    comprehensive analysis and presented exhaustive details                a different project.
    to justify the substantial investments of our clients. But we
    ignored or sacrificed the practical in our dedication to a
    highly principled approach.




3                                                                                                                      © Sapient Corporation 2011
                                                                     21st Century User Testing: Test It Cheap, Fast and Often




    DOING LESS                                                              Test with three people and recruit loosely
    When a client needs usability testing, the package above                Instead of trying to find all possible issues with no concern
    is what we typically suggest. Depending on the client,                  for how many we can actually address, our revised, more
    project and budget, we might get asked for a less ambitious             practical goal is to concentrate on the most serious
    approach and the cheaper we need to go, the more guerilla               usability problems. These tend to be about navigation, page
    we get with our tactics. We find ways to get usability testing          layout, and other issues that don’t require participants with
    done on-site and we find ways to do things on a smaller                 domain knowledge, so recruitment can be less precise.
    scale. We assure our client that the reduced scope will still           Three participants a month will surface the significant
    result in credible data, but we’re almost apologetic about              problems and running so few sessions makes it possible to
    our methods.                                                            have both the testing and the debriefing on the same day.

    But to unleash usability testing’s true potential, we must              As with traditional methods, testing is one-on-one with
    now adopt our guerilla tactics as the norm rather than                  sessions lasting up to an hour. A facilitator in the same
    mumble about them in embarrassment. Steve Krug,                         room follows a script, asking each participant to work
    author of Don’t Make Me Think and Rocket Surgery Made                   through the same series of tasks and asking them to think
    Easy, provides a blueprint for a more practical approach to             aloud as they do so.
    usability testing.
                                                                            Conduct usability testing on-site and make it
    Test whatever you have at the same time                                 an all-inclusive spectator sport
    every month                                                             Testing at off-site facilities automatically limits the number
    Testing should start as early in the project as possible, even          of people who can attend. By moving it to the client’s offices
    earlier than you think makes sense. Organizations have                  (or to space extremely convenient to the client) we can
    historically done usability testing in a single, exhaustive             involve every member of the team, every stakeholder and
    round at the end of the development cycle. While this                   everybody in the client’s organization with an interest in the
    approach generates a high volume of data, little of it is               project. Watching users struggle with tasks transforms the
    useful because it arrives too late in the development cycle.            people who see it and will generate the support we always
    By committing to monthly testing, we can change the                     expected (but never got) from old-school reports.
    culture of the project and its relationship to research.
                                                                            Testing takes place in one room as the audience watches
    It’s always tempting to wait for the thing you’re building              in another. The two rooms should be far enough apart so
    to get to critical mass before testing it. But with testing             that the test participants remain unaware of their audience.
    materials required in advance, the data can already be                  There are no special requirements for the rooms (they can
    weeks old before it’s available to the team (assuming that              be regular offices or conference rooms). The room where
    testing takes at least a week and analysis takes at least               the testing takes place should have a computer with Internet
    another week). Also, research results served in one or two              access, a monitor, a mouse, and a keyboard. The computer
    bulky servings (as it is with traditional methods) is harder            should have screen recording software (e.g., Camtasia) and
    for an organization to digest than data in regular, smaller             screen sharing software (e.g., WebEx) so the audience in
    servings delivered over the course of the project.                      the second room can watch in real time what’s happening
                                                                            on the test participant’s screen. An audio solution as simple
                                                                            as a telephone set on speakerphone will be needed so the
                                                                            audience can hear the test participant (although the test
                                                                            participant should never hear the audience).




4                                                                                                                      © Sapient Corporation 2011
                                                                      21st Century User Testing: Test It Cheap, Fast and Often




    Get everybody who watches the sessions to                                Improving usability almost always requires taking things
    come to the debriefing                                                   away rather than adding things. The solutions we are
                                                                             looking for should be as simple and unambitious as
    After completing the three sessions and sending the test
                                                                             possible, because we have to implement them before the
    participants away, the team discusses what they saw over a
                                                                             next month’s regularly scheduled testing.
    lunch that can be as simple as pizza and sodas. Each team
    member brings with them a list of the three most important
    usability problems they saw for each session. By socializing
    the research, each member of the team becomes a source                   CONCLUSION
    of data. Let people talk about whether they thought it was               Instead of being embarrassed by our guerilla tactics,
    important if a test participant did or did not click a button.           we should be recommending their use as a regular part
    Let them have that argument along with any others that                   of every engagement. If a client is determined to pay big
    come up. If you don’t have the facilitation skills to maintain           money for traditional usability testing, we should do it and
    a healthy and productive conversation, bring the talent                  do it well. We provide usability testing as a service, but it’s
    in. (There is usually someone on your team that can fill                 not what we’re selling when we make our pitch. What we’re
    in as a facilitator.) By making usability testing a spectator            selling, what we deliver, is our client’s success.
    sport, research results can be internalized by the whole
    organization.                                                            Regularly scheduled, small-scale, highly practical usability
                                                                             testing will deliver success and change our clients’
    Focus ruthlessly on only the most serious                                organizations. The dirty little secret of 21st Century usability
    problems.                                                                testing is that if we can’t talk the client into paying for it, we
    A successful debriefing session will result in a list of the             should do it anyway and Sapient should pick up the tab. It’s
    most serious usability problems encountered, along with a                smart and cheap and it greatly increases our ability to create
    list of the problems that will be fixed before the next month’s          wildly successful interfaces, applications, and services.
    round of testing. Severity will be gauged by considering how
    many people will experience an issue and whether it will                 It’s time for a new approach to usability testing. By focusing
    cause a serious problem for the people who experience it                 on the practical application of user testing data rather than
    or if it will just be an inconvenience.                                  clinging to principles that impractically aspire to perfection,
                                                                             the practice of usability testing will finally reach its potential
    Failing to deal with a serious problem ruthlessly guarantees             in the first part of the this new century.
    that the problem will show up in future testing. It takes a
    severe ruthlessness to identify a serious problem and make
    it go away.

    As a follow-up to the debriefing, we send a short e-mail
    that briefly covers what was tested, the list of tasks the
    participants performed, and the list of problems to be
    fixed before the next test. The e-mail should take less
    than two minutes to read (making it more likely that
    most stakeholders will read it). In addition, it should be
    delivered in a format readable from any device (which
    means stakeholders can read it wherever and whenever
    they prefer). This e-mail replaces the voluminous report
    delivered and presented at the end of a traditional usability
    testing engagement.




5                                                                                                                          © Sapient Corporation 2011
                               21st Century User Testing: Test It Cheap, Fast and Often



    Subject Matter Expert: Dan Willis
    Dan Willis is an Associate Creative Director for Sapient Government Services. He has been designing
    web sites since the mid-1990s. Dan is prominent in the user experience community in the U.S., having
    presented at the last two South by Southwest Interactive Festivals as well as numerous Usability
    Professionals’ Association, Information Architecture Institute, and Interaction Design Association
    conferences. He is also the creator of uxcrank.com, a resource for UX professionals.




6                                                                                © Sapient Corporation 2011