How many users should you test with? The 5-user rule

In short

How many users you test with depends on what you want from the test. If you are looking for problems, 5 people are enough. If you want to measure a number reliably, even 20 people is the minimum.

The test itself, writing tasks and running sessions, is covered in the usability testing guide. This page looks only at sample size: where does the 5 come from, and when does it mislead?

• The 5-user rule is based on a model. According to it, about 85% of the problems show up with the first five people.
• The rule works with repeated small tests. If your users are very different or you are measuring a number, it doesn’t help.

Where does the 5-user rule come from?

The rule comes from Jakob Nielsen’s text “Why You Only Need to Test with 5 Users” of March 18, 2000. Nielsen bases the number on a mathematical model he published with Thomas K. Landauer in 1993.

According to the model, the number of problems found with n users is:

N × (1 − (1 − L)ⁿ)

N is the total number of problems in the design. L is the share of problems a single person finds. According to Nielsen, the typical value of L, averaged over many projects studied, is 31%.

Why does the curve bend at 5?

The first person shows about a third of the problems. The second repeats some of them but brings new ones too. With each additional person the repetition grows, and new findings become rarer.

With L = 31%, the formula gives 84.4% for 5 people. Nielsen rounds that to 85% in his text. He also names the start of the curve plainly: zero users, zero insight.

Nielsen writes that after the fifth person you are wasting time. According to the same text, it takes at least 15 users to find all the problems. Even so, he doesn’t recommend seeing those 15 people in a single test.

The real recommendation: 3 × 5

Nielsen recommends splitting the budget for 15 people into 3 tests with 5 each. For him, the goal of a usability study is to improve the design. Merely documenting weaknesses isn’t enough.

You fix the problems from the first test. The second test finds most of the remaining 15% and shows whether the fixes work. A new design sometimes brings new problems too, and you only see them if you test again.

Draw the curve yourself

Below you can change two values: the number of users and the share one person finds. The dashed line is the curve with Nielsen’s average of 31%. Notice how far your own estimate departs from that line.

Sample size calculator

84%of problems found

• Missed: 16%
• Contribution of the last person added: 7%
• Needed for 90%: 7 people, for 95%: 9 people

Default values: 5 users, L = 31%.

• 1 ppl31%
• 2 ppl52%
• 3 ppl67%
• 4 ppl77%
• 5 ppl84%
• 6 ppl89%
• 8 ppl95%
• 10 ppl98%
• 12 ppl99%
• 15 ppl99.6%
• 20 ppl99.9%

The values are calculated with Nielsen and Landauer’s formula 1 − (1 − L)^n and rounded. I chose the range of 5 to 60 for L. The model assumes that similar users use the product in similar ways.

Lower L to 15% and leave 5 users. The share found drops to about 56%, and 90% takes 15 people. Here lie the rule’s strength and weakness: everything depends on L.

Look at the line “Contribution of the last person added” too. At L = 31%, the fifth person adds about 7% new share, the tenth about 1%. That is exactly the diminishing return.

Why isn’t L 31% in every project?

31% is the average of many projects. The L in your product can differ markedly. The text gives no range for how much L varies from project to project.

The formula also has a precondition. According to Nielsen, the model only applies to users who resemble each other and use the site in fairly similar ways. On a marketplace, buyers and sellers don’t meet this condition.

My observation: L drops with the breadth of the product and tasks. When I test a single payment flow, participants get stuck in the same places. When I test ten different screens of a B2B panel, each person ends up in a different corner, and the problems overlap less.

That is why I use the 5 for a single flow and a single user group. When the scope gets broader, I split the test into flows.

When are 5 users not enough?

NN/g’s own texts draw the limits of the 5 clearly. The table below sums up the recommendations from three texts.

Type of study	Number recommended by NN/g	Source
Qualitative usability test (finding problems)	5 users	Nielsen 2000, 2012
Two different user groups	3–4 per group	Nielsen 2000, 2012
Three or more groups	3 per group	Nielsen 2000
Quantitative study (metrics such as time, success rate)	At least 20	Nielsen 2006, 2012
Card sorting	At least 15 per group	Nielsen 2012
Eyetracking, for stable heatmaps	39	Nielsen 2012

Different user groups

If your product targets very different people, test each group separately. Nielsen recommends 3 to 4 people per group for two groups and 3 per group for three or more groups. He adds that it takes at least 3 people to see the variety within a group.

The more groups, the fewer people per group, because behaviour overlaps between groups. If you have personas, start by defining the groups. Only separate groups that really use the product differently.

Quantitative studies

If you measure the time for a task or the success rate, 5 people aren’t enough. Nielsen’s 2006 text recommends 20 users for such studies and justifies it with the confidence interval.

The example in the text uses a 90% confidence interval. With 10 users, the margin of error is ±27% of the mean. If you remove one outlier from 20 people and keep 19, the margin of error drops to ±19%. Nielsen considers this interval sufficient for most purposes.

To bring the margin of error down to ±10%, you need data from 71 users. Because you discard outliers, you test with 76 people. According to Nielsen, that is a waste of money in almost all practical projects. According to the same text, a quantitative study costs about four times as much as a qualitative test with 5 people.

Nielsen recommends making the first studies qualitative. Comparing a survey score such as the SUS score is also a measurement. An average from a few people has a wide margin of error.

Card sorting and eyetracking

Nielsen’s 2012 text recommends at least 15 users per group for card sorting. Card sorting shows, when building the information architecture, how users group content. For stable heatmaps in eyetracking, the same text asks for 39 users.

Criticism: Faulkner 2003

The most cited criticism of the 5-user rule is Laura Faulkner’s 2003 article. It appeared in the journal Behavior Research Methods, Instruments, & Computers.

According to the abstract, Faulkner tested with 60 users. She then drew random subgroups of 5 or more people from this group. Some groups of 5 found 99% of the problems, some only 55%.

With more people, the worst case got better. With 10 users the lowest discovery rate rose to 80%, with 20 users to 95%.

Does that refute Nielsen?

In my view, the two studies answer different questions. Nielsen describes the average: a typical test with 5 people finds most problems. Faulkner shows the risk: a single test with 5 people can, depending on luck, also find very little.

Nielsen’s 3 × 5 recommendation partly absorbs this risk. A problem that slips through in the first round shows up in the second or third. If, on the other hand, you run only one test and tick the product off as “tested”, Faulkner’s warning weighs heavily.

Nielsen’s 2012 update

Nielsen returned to the topic in 2012. At an NN/g conference he asked 217 participants about practice in their companies. The average answer was 11 users per round. According to Nielsen, that is more than double the recommended number.

The same text leaves room in both directions. For projects with little overhead, even 2 users per study can be the best number. In consulting projects, on the other hand, NN/g often works with about 8 users. According to Nielsen, the consulting nature justifies a few more users.

The text also corrects a misconception: a large total number of users doesn’t change the sample needed. Even for a banking app with millions of users, 5 people in a qualitative test are a valid start.

How to choose your own number

When I plan a test, I don’t pick the number from a table. I first answer the questions below, and the number follows.

Are you looking for problems or measuring a number? If you are looking for problems, build a qualitative test and start with 5 people. A question such as “Is the new checkout faster than the old one?”, on the other hand, can only be answered by a quantitative study.

How many different user groups do you have? In a food ordering app, ordering customers and restaurant staff don’t see the same screens. Two groups mean two small tests.

What does the test cover? A single flow is a narrow scope; participants get stuck in the same places. A whole app is broad. With a broad scope, either increase the number of people or split the test into flows.

Is there a next round? If you test again after the fixes, 5 people per round are enough. If you have only one chance, remember Faulkner’s finding and increase the number.

Whatever number you choose, plan a few backup participants. In a test with 5 people, one no-show is a fifth of the sample.

My note

The most common mistake I see in teams: they don’t test at all. The worry “The results won’t be statistically significant” postpones the first test by months.

Finding problems doesn’t need significance, though. A button where one person gets stuck three times is evidence enough to fix it.

For a flow I start with 5 people and watch two things. If the last two participants show no new problem, I stop and fix. If new problems keep appearing, L is low. Either the flow is too broad or the participants don’t resemble each other.

If your budget doesn’t stretch to even one test round, you can start with an expert review. A heuristic evaluation doesn’t replace a user test. But it shows which flow to test, so you don’t waste the 5 people’s time on obvious problems.

If you want several flows of your product reviewed at once: UX Audit Pro reviews up to 10 flows end to end. The priority list from it also tells you where to run your first test with 5 people.