21st Century User Testing: Test It Cheap, Fast and Often
21st Century User Testing: Test It
Cheap, Fast and Often
1 © Sapient Corporation 2011
21st Century User Testing: Test It Cheap, Fast and Often
21st Century User Testing: Test It
Cheap, Fast and Often
Four decades of usability testing has done little to make our believing that we can somehow create reports that are so
world any more usable. This is a shame because usability effective that stakeholders find them irresistible. But that’s
testing is such a unique and powerful development tool. like believing there’s an umbrella so effective that it can
By watching people work through specific tasks using the stop rain from falling.
interfaces, applications and services we build, we avoid our
own biased views of what people want and need. It provides We kill ourselves to catalogue every possible usability issue,
a much greater chance for us to meet users’ requirements but we need to step back and ask, how many issues from
and align with their expectations. such an exhaustive list actually get resolved in a typical
project? 20 percent? If we got to even half of the issues
You would expect the use of such an impressive tool for 40 raised by testing, it would be a miracle.
years to have a positive influence on the overall usability of
all solutions. So why hasn’t this been the case? For years, we had long debates about the appropriate
number of usability test participants until finally the
For one thing, clients may listen intently when we present our research determined that 85 percent of problems surfaced
findings, but the data doesn’t seem to stick. In organizations with five test participants, and we all nodded our heads and
with weak project management, the data doesn’t stick felt secure in our methods. But what practical value have
because no one is directly responsible for acting on we seen in 85 percent discovery when we have only been
usability testing results. In organizations with strong project able to address a handful of issues?
management, the data doesn’t stick because—whether or
not it fits the project manager’s job description—everyone Even with all of its flawed elements, traditional practices
assumes that the project manager will integrate research have given us glimpses of usability testing’s true potential.
results into projects. This means that no one other than the User Interface Engineering (UIE), a leading user research
project manager takes personal responsibility. firm, tested an e-commerce Web site for a major corporate
client where first-time buyers encountered a screen similar
Another influence on usability testing’s lack of long-term to the one in Figure 1.
impact is the way we’ve traditionally reported results. We
craft the most comprehensive analysis and communication
of our research possible, but despite our efforts people
rarely read the report. Many of us have taken this personally,
seeing it as a reflection on the quality of our work and
2 © Sapient Corporation 2011
21st Century User Testing: Test It Cheap, Fast and Often
There is no magic that will make traditional usability testing
data suddenly stick within an organization, no secret formula
for creating reports that stakeholders will actually read, and
no new process that will allow projects to address a majority
of issues raised. The only real answer is to radically change
how we do usability testing, with the most radical part being
that we actually need to be doing less.
THE WAY WE’VE ALWAYS DONE USABILITY
Figure 1. The Login Screen with a Register Button
TESTING
Participants resented being forced to register before
making a purchase. As one person said, “I’m not here to This is how it typically works. You pay a recruiter to find
enter into a relationship, I just want to buy something.” The people similar to your users. You rent a facility and invite
client’s intent had been to make shopping easier, but their key stakeholders to observe the test. Your test participants
solution actually made it much more difficult. Based on the work their way through a series of tasks using a fully- or
usability testing results, the client changed the “Register” semi-functional version of the Web site or application your
button to a “Continue” button (as shown in Figure 2). This team is building.
made it more obvious that registering was optional.
You review the video of each session again and again
until you’ve gleaned every possible data point about every
possible usability issue. You combine, compare, and classify
all of the data and; issues are prioritized and placed into a
comprehensive catalog of concerns.
You take screenshots from the interface and snapshots
of the testing and combine them with your insightful,
exhaustive analysis. Then you deliver it all to the client in
one giant, glorious report.
Figure 2. The Login Screen with a Continue Button
Starting the day they changed the button, the number In a meeting you present your data and watch as your client
of customers purchasing jumped 45 percent. In the first scrawls notes on the report cover. After skimming the
month, the client made $15 million more than it had in the executive summary, the client tosses your shiny, full-color,
month before the button was changed. In the first year, it indexed report on a shelf with a stack of others, and she
made $300 million more in sales than it had the year before. buries the digital version deep in her hard drive. Your keen
insights and analysis eventually blend in with the existing
Despite hundreds of thousands of usability tests and white noise of her organization’s developmental mythology.
projects as successful as the $300,000 button example,
the quality of usability across all products and interfaces If you’ve presented well and stayed within your budget—
hasn’t improved. It certainly wasn’t the result of a lack of usually between $20,000 and $40,000, depending on the
effort. We strove to find the right number of participants to number of types of users and your team’s travel costs—you
surface the greatest number of issues. We carefully crafted get the chance to do it all over again for a later phase or for
comprehensive analysis and presented exhaustive details a different project.
to justify the substantial investments of our clients. But we
ignored or sacrificed the practical in our dedication to a
highly principled approach.
3 © Sapient Corporation 2011
21st Century User Testing: Test It Cheap, Fast and Often
DOING LESS Test with three people and recruit loosely
When a client needs usability testing, the package above Instead of trying to find all possible issues with no concern
is what we typically suggest. Depending on the client, for how many we can actually address, our revised, more
project and budget, we might get asked for a less ambitious practical goal is to concentrate on the most serious
approach and the cheaper we need to go, the more guerilla usability problems. These tend to be about navigation, page
we get with our tactics. We find ways to get usability testing layout, and other issues that don’t require participants with
done on-site and we find ways to do things on a smaller domain knowledge, so recruitment can be less precise.
scale. We assure our client that the reduced scope will still Three participants a month will surface the significant
result in credible data, but we’re almost apologetic about problems and running so few sessions makes it possible to
our methods. have both the testing and the debriefing on the same day.
But to unleash usability testing’s true potential, we must As with traditional methods, testing is one-on-one with
now adopt our guerilla tactics as the norm rather than sessions lasting up to an hour. A facilitator in the same
mumble about them in embarrassment. Steve Krug, room follows a script, asking each participant to work
author of Don’t Make Me Think and Rocket Surgery Made through the same series of tasks and asking them to think
Easy, provides a blueprint for a more practical approach to aloud as they do so.
usability testing.
Conduct usability testing on-site and make it
Test whatever you have at the same time an all-inclusive spectator sport
every month Testing at off-site facilities automatically limits the number
Testing should start as early in the project as possible, even of people who can attend. By moving it to the client’s offices
earlier than you think makes sense. Organizations have (or to space extremely convenient to the client) we can
historically done usability testing in a single, exhaustive involve every member of the team, every stakeholder and
round at the end of the development cycle. While this everybody in the client’s organization with an interest in the
approach generates a high volume of data, little of it is project. Watching users struggle with tasks transforms the
useful because it arrives too late in the development cycle. people who see it and will generate the support we always
By committing to monthly testing, we can change the expected (but never got) from old-school reports.
culture of the project and its relationship to research.
Testing takes place in one room as the audience watches
It’s always tempting to wait for the thing you’re building in another. The two rooms should be far enough apart so
to get to critical mass before testing it. But with testing that the test participants remain unaware of their audience.
materials required in advance, the data can already be There are no special requirements for the rooms (they can
weeks old before it’s available to the team (assuming that be regular offices or conference rooms). The room where
testing takes at least a week and analysis takes at least the testing takes place should have a computer with Internet
another week). Also, research results served in one or two access, a monitor, a mouse, and a keyboard. The computer
bulky servings (as it is with traditional methods) is harder should have screen recording software (e.g., Camtasia) and
for an organization to digest than data in regular, smaller screen sharing software (e.g., WebEx) so the audience in
servings delivered over the course of the project. the second room can watch in real time what’s happening
on the test participant’s screen. An audio solution as simple
as a telephone set on speakerphone will be needed so the
audience can hear the test participant (although the test
participant should never hear the audience).
4 © Sapient Corporation 2011
21st Century User Testing: Test It Cheap, Fast and Often
Get everybody who watches the sessions to Improving usability almost always requires taking things
come to the debriefing away rather than adding things. The solutions we are
looking for should be as simple and unambitious as
After completing the three sessions and sending the test
possible, because we have to implement them before the
participants away, the team discusses what they saw over a
next month’s regularly scheduled testing.
lunch that can be as simple as pizza and sodas. Each team
member brings with them a list of the three most important
usability problems they saw for each session. By socializing
the research, each member of the team becomes a source CONCLUSION
of data. Let people talk about whether they thought it was Instead of being embarrassed by our guerilla tactics,
important if a test participant did or did not click a button. we should be recommending their use as a regular part
Let them have that argument along with any others that of every engagement. If a client is determined to pay big
come up. If you don’t have the facilitation skills to maintain money for traditional usability testing, we should do it and
a healthy and productive conversation, bring the talent do it well. We provide usability testing as a service, but it’s
in. (There is usually someone on your team that can fill not what we’re selling when we make our pitch. What we’re
in as a facilitator.) By making usability testing a spectator selling, what we deliver, is our client’s success.
sport, research results can be internalized by the whole
organization. Regularly scheduled, small-scale, highly practical usability
testing will deliver success and change our clients’
Focus ruthlessly on only the most serious organizations. The dirty little secret of 21st Century usability
problems. testing is that if we can’t talk the client into paying for it, we
A successful debriefing session will result in a list of the should do it anyway and Sapient should pick up the tab. It’s
most serious usability problems encountered, along with a smart and cheap and it greatly increases our ability to create
list of the problems that will be fixed before the next month’s wildly successful interfaces, applications, and services.
round of testing. Severity will be gauged by considering how
many people will experience an issue and whether it will It’s time for a new approach to usability testing. By focusing
cause a serious problem for the people who experience it on the practical application of user testing data rather than
or if it will just be an inconvenience. clinging to principles that impractically aspire to perfection,
the practice of usability testing will finally reach its potential
Failing to deal with a serious problem ruthlessly guarantees in the first part of the this new century.
that the problem will show up in future testing. It takes a
severe ruthlessness to identify a serious problem and make
it go away.
As a follow-up to the debriefing, we send a short e-mail
that briefly covers what was tested, the list of tasks the
participants performed, and the list of problems to be
fixed before the next test. The e-mail should take less
than two minutes to read (making it more likely that
most stakeholders will read it). In addition, it should be
delivered in a format readable from any device (which
means stakeholders can read it wherever and whenever
they prefer). This e-mail replaces the voluminous report
delivered and presented at the end of a traditional usability
testing engagement.
5 © Sapient Corporation 2011
21st Century User Testing: Test It Cheap, Fast and Often
Subject Matter Expert: Dan Willis
Dan Willis is an Associate Creative Director for Sapient Government Services. He has been designing
web sites since the mid-1990s. Dan is prominent in the user experience community in the U.S., having
presented at the last two South by Southwest Interactive Festivals as well as numerous Usability
Professionals’ Association, Information Architecture Institute, and Interaction Design Association
conferences. He is also the creator of uxcrank.com, a resource for UX professionals.
6 © Sapient Corporation 2011