What is usability testing, and how do you run it?

In short

In a usability test you sit a user in front of your product, give them a task and stay quiet. While they try to complete the task, you watch and take notes. At the end you don’t have opinions in hand, but observations.

According to NN/g’s introductory guide, most tests have three parts: a facilitator, tasks and participants. The facilitator guides the session and tries not to influence the participant’s behaviour without meaning to. The tasks are things participants might do in real life. Participants are realistic users of the product or people with a background similar to the target group.

What the test measures is usability: how easy a product is to learn and use. The test reads that ease from user behaviour.

The difference from heuristic evaluation and a UX audit

In a heuristic evaluation there are no users; experts review the interface against principles. A UX audit adds accessibility and content checks. NN/g’s guide to heuristic evaluation says it clearly: this method doesn’t replace user research. I use both, one after the other. The audit tells me where to look; the test shows whether users really get stuck there.

When to test

The short answer: while design changes are still cheap. You can test a paper sketch, a clickable prototype or the live product. According to Nielsen, thinking aloud is a flexible method too. It can be used at any stage of development and with most kinds of interfaces.

I recommend a test at these moments:

• Before a flow is built, at the prototype stage. A problem found here is fixed in an afternoon.
• Shortly before a release, for the two or three most important tasks.
• When analytics show the drop-off rate of a step rising. The numbers say where people drop off; the test shows why.
• Before a redesign, with the old product. Drawing the new one without knowing what works throws away the good parts too.

A test doesn’t answer every question. “Do users want this feature?” is a question for interviews or market research. The test checks whether an existing design can be used. It doesn’t tell you whether an idea for an MVP has demand.

Types of usability tests

I tell tests apart with two questions: does someone facilitate the session, and are you in the same room as the participant? The third distinction is the purpose. A test either looks for problems or collects numbers.

Moderated and unmoderated

In a moderated test, the facilitator is with the participant throughout the session. According to an article by Amy Schade at NN/g, the facilitator can dig into problems with follow-up questions after the tasks. They can also gently get a quiet participant to say more. Asking “What did you expect there?” after a task brings information you can’t read from the recording.

In an unmoderated test, the participant does the study alone. The session is recorded and you watch it later. Questions written in advance can be added; real-time follow-up questions can’t. Schade finds this type suitable when the study focuses on a few specific elements and time is short.

The price of an unmoderated test is a loss of control. Follow-up questions are written in advance and asked of everyone the same way. If the task text is bad, you only notice after the session. That is why the task text matters twice as much in this type.

Remote and in person

A moderated remote test is very similar to an in-person test. The difference: facilitator and participant are in different places. According to Schade, the facilitator communicates by phone, email, chat or a mix of these. According to NN/g, a remote test often costs less time and money than an in-person study.

In an in-person test you see the participant’s face and their hesitation. Schade also prefers the in-person test as a general rule. But she adds that a remote test is better than no test at all when budget or time is short.

One more practical tip for remote work: the same article recommends recruiting a few more people, because no-show rates in remote studies can be higher. Silence on the other end is ambiguous too. According to Schade, someone may be silent because they are confused, distracted or immersed in the content.

Qualitative and quantitative

NN/g’s distinction is clear. A qualitative test is best for finding problems in the experience. A quantitative test collects metrics that describe the experience, such as task success or time on task. It is best for benchmarking. This page mainly describes the qualitative test, because product teams need it most often.

How to run a usability test: step by step

I build a test in seven steps. The order matters: no task without a written goal, no participant selection without written tasks.

1. Write down the goal

Write in one sentence what you want to learn from the test. “Can guest users complete the payment step without help?” is a good goal. “Is our website user-friendly?” can’t be answered.

Next to the goal I write who uses the result and which decision it influences. If there is no decision, there is no need for a test.

2. Choose the tasks

Choose tasks that serve the goal and that users really do. I don’t put more than a few tasks into one session. Someone who gets tired loses attention in the last tasks.

Write a success criterion for each task. A clear end point such as “The participant has reached the payment screen” prevents arguments in the analysis. You don’t tell the participant this criterion; only you know it.

3. Find participants

Participants are real users of the product or people with the same needs. Having a colleague from the design team test a banking app tells you almost nothing.

Participants get an incentive for their time. According to NN/g’s introductory guide, these payments together usually cost a few hundred dollars for a simple study. I prefer something participants really use, such as a gift card.

If your product serves very different groups, take separate participants from each group. How many you need is covered below.

4. Write the scenario

The scenario puts the task into a story: why is the participant doing this? Marieke McCloskey’s article at NN/g gives three tips. The task is realistic, asks the participant to do something, and contains neither clues nor step descriptions. The article also says the scenario should give context.

The most useful warning in the article: don’t use words in the task that appear in the interface. Instead of “Sign up for the newsletter”, it says “Make sure news about events arrives in your inbox regularly”. The generator below shows this difference for each type of product.

Try the tasks with a colleague before the first real participant. One confusing sentence can waste five sessions at once.

5. Ask for thinking aloud

You ask the participant to say out loud what goes through their mind while they work. In his 2012 article, Nielsen calls thinking aloud perhaps the single most valuable usability engineering method. He boils the method down to three steps: recruit representative users, give them representative tasks, and keep quiet.

Keeping quiet is the hardest part. It is also one of the weaknesses Nielsen names: an untrained facilitator who steps in can easily change the user’s behaviour. When the participant gets stuck, I ask “What are you trying to do right now?”, not “Did you look at the top right?”.

Know this too: the situation isn’t natural. According to Nielsen, many people find it hard to talk continuously, and some filter what they say. A short reminder to a quiet participant is enough.

I run the session in the same order every time. First I tell the participant that they aren’t being tested; the product is. I add that there is no wrong answer and that every place they get stuck helps us.

Then I listen, with a few warm-up questions, to how they use the product. I give the tasks one at a time and in writing. At the end I ask for their overall impression and whether anything else comes to mind. Putting opinions at the end doesn’t disturb behaviour during the tasks.

6. Take notes

Taking notes means writing down what you see, not what you think. “Didn’t find the price filter, scrolled up three times” is an observation. “The filter is badly designed” is an interpretation and belongs in the analysis.

For each task I fill in the same lines: successful or not, where they hesitated, what they said. If possible I record the session, and someone from the team takes notes separately. Whoever facilitates and writes at the same time misses something.

7. Analyse

Once the sessions are over, turn the notes into problems. Group participants who got stuck at the same place under one problem and write down how many there were. A problem from a single participant goes on the list too, but with that note.

Then rank each problem by severity. Here I use the same 0 to 4 scale as in the UX audit. The report is short: the most important problems, a video clip as evidence for each and a concrete suggestion for the fix.

How many participants you need

For a qualitative test meant to find problems, five participants are usually enough. The number comes from Jakob Nielsen’s 2000 article “Why You Only Need to Test with 5 Users”. But the article’s real recommendation isn’t to settle for five; it is to repeat small tests.

Nielsen bases the number on a model he published with Thomas Landauer at the INTERCHI conference in 1993. According to the model, the number of problems found with n users is N(1 − (1 − L)ⁿ). N is the total number of problems in the design, L the share a single user finds.

In the projects Nielsen studied, L averaged 31%. A single user therefore shows almost a third of the problems. A first study with five participants finds about 85% of the problems. According to the article, with each additional user you see the same things again and again and learn less and less.

• 1 person31%
• 2 people52%
• 3 people67%
• 4 people77%
• 5 people84%
• 6 people89%
• 8 people95%
• 10 people98%
• 15 people99.6%

Figure: shares from Nielsen’s formula with L = 31%. The values are rounded.

The formula gives 84.4% for five people; the article rounds it to about 85%. Notice how the curve flattens after five. The fifteenth person shows almost no new problem.

Nielsen concludes: to find all the problems you need at least 15 users, but don’t spend that budget on one test. Run three tests with five people each. After the first test, fix the design. The second test checks the fixes and finds most of the problems the first round missed. The third is needed for deep problems that only become visible once the surface problems are gone.

The limits of the number five are in the same article:

• If your users fall into very different groups, such as buyers and sellers, take separate participants from each group. Nielsen recommends 3 to 4 per group for two groups and 3 per group for three or more groups.
• For quantitative studies, that is, when you collect numbers, the article points to a later NN/g recommendation: 20 users.
• For card sorting, the same article gives 15 users.

NN/g’s 2019 introductory guide is in line with this too: a typical qualitative study is done with five participants.

Generator for tasks and scenarios

The quality of a test is mostly decided by the task text. Written well, the same goal shows the user’s real behaviour. Written badly, it whispers the answer to the participant.

Building tasks and scenarios

Online store · Purchase or payment

Good Task read to the participant

Your daughter’s birthday is next week. You want to buy her headphones that don’t break your budget. The gift should arrive before Saturday. Find a suitable product and go as far as the payment step.

Success criterion (your note only): A product that fits the budget and delivery date is in the cart, and the participant is on the payment screen.

• Realistic A task that can happen in the participant’s life.
• Asks to do Doesn’t ask them to tell, but to do the task on the screen.
• No hint No menu name and no button text.

Bad The same goal, badly written

Type “headphones” in the search bar and press “Add to cart”. Then tell me whether you like the payment page.

• Names the steps If the path is described, the test can’t measure whether the interface shows the path.
• Names the interface label The participant learns the word to look for from the task. Finding that word on the screen is easier than understanding the task.
• Asks for an opinion A question about liking shows no behaviour. You want to see where the participant gets stuck while doing it.

Choose the type of product and the user’s goal. The left card shows a good task text read to the participant; the right one shows the same goal badly written and why it is bad. The tasks are my examples, not from a real client project.

Most bad examples break at the same place: whoever writes the task knows the screen by heart. If you see a menu name or button text in your task, replace it with the word the user would use themselves.

Common mistakes

Interface words in the task. The participant reads the task, looks for the same word on the screen and finds it. The test looks successful but has measured nothing.

Asking them to tell instead of do. McCloskey recommends letting the participant do the task rather than describe how they would do it. Their own account is less accurate and hides how easy or frustrating the interface is.

Participants from your own team. People who know the product don’t see the problems. Participants should resemble real users.

Helping the participant. Rescuing a participant who is stuck is a natural reflex. But the moment you help, the data for that task is lost. Ask first: “What did you expect here?”

One big test. A test with twenty people means seeing the same problem twenty times. Nielsen’s recommendation is to repeat small tests with five people each.

Mixing observation and interpretation. Instead of “was confused”, write down what they did. Interpretation happens in the analysis, across several sessions.

Burying the results in a report. A long report nobody reads wastes the test. A short list of problems and a few video clips get the team moving faster.

Testing a mobile product on desktop. If your users come from phones, test on phones. Where the thumb can reach brings problems to light that can’t be seen on desktop at all.

My note

The step teams find hardest isn’t writing the tasks but keeping quiet in the session. When the user gets stuck, the first urge is to show them the way. In those moments I count to five in my head and then only ask what they expected.

My second observation: the most valuable output of the first test is often the mistakes in the task texts. So don’t skip the pilot session. Half an hour of rehearsal with a colleague saves the real sessions.

If your budget doesn’t stretch to a test, first find the riskiest flow with an audit. Spend the test budget on that flow. For a look across the whole product, there is the UX Audit Pro.