Updated: Nov 30, 2021
There are dearly-held beliefs in most disciplines, and user experience design is no different. UX is known for its numerous best practices, and rules that are to be used as guidelines when offering advice, strategy, and direction to clients.
One of the most-often cited rules is that a user test only requires 5 or so users to uncover 80% of the usability problems in an interface for websites, apps or SaaS. Heck, I’ve even said it myself, although I had a nagging voice in the back of my head wondering if that were true.
As Jared Spool pointed out in a recent article entitled, "A Fundamental Mind Shift for Usability Testing," the rule comes from luminaries the Nielsen-Norman group, and a paper by Bob Virzi published in 1990 called, “Streamlining the Design Process: Running Fewer Subjects.” To quote from the paper directly:
“The basic findings are that (1) with between 4 and 5 subjects, 80% of the usability problems are detected and (2) that additional subjects are less and less likely to reveal new information.”
Turns out this rule was derived from looking at 5 user testing studies a loooong time ago. Think about that: 5 studies (A TINY NUMBER) were used to draw this conclusion well before the Internet era exploded. Is this conclusion statistically significant? Turns out, it’s not.
This begs the question: “How many users DO we need to conduct comprehensive user testing?” It’s not wrong to ask that question, but perhaps it’s also not right.
Maybe we should get more curious:
Why are design professionals so quick to buy into this thinking? And what about various user types who use a digital product—wouldn’t different customers have different needs? How are we making sure we cover all of those users? And what kinds of user testing and user research are the most useful? Is this the best way to test and learn?
What if it’s more important to test ideas and concepts, and research new ways of solving problems than trying to catalogue all of the flaws in an existing product or service?
Jared Spool talks a lot about design maturity. It turns out that post-launch user testing is, well, kind of basic. There are many research approaches an organization can take earlier in the design process other than looking at what they already have once it’s done.
We should think more broadly about how to create products and services that people will want to adopt, and how to design them so that they are customer-informed, customer-focused and easy to use.
If you want to have millions of customers, talk to as many customers as you can:
Conduct research earlier than you think you need to—start by using techniques like surveys and contextual inquiries to pinpoint needs, remotely, and in person
Test concepts, and prototypes before you bring anything to market
Use online research tools that allow you to reach scores if not hundreds of test subjects, especially early on
Refine the concepts, and prototypes, and test againMonitor user behavior after launch, and refine as needed
User testing at or near the end of a project with 5 users is a waste of time.
Your first line should be dedicated user experience resources deployed on an ongoing basis to innovate, and scale quickly, doubling down on ideas that work, and discarding those that don’t.
Subscribe to our bi-monthly email newsletter for the latest insights from our work.