Heuristic evaluation: what it is and how to run one

In short

In a heuristic evaluation you go through an interface screen by screen with a short list of principles in hand. Those principles are called heuristics: rules of thumb, not strict laws.

Jakob Nielsen and Rolf Molich published the method in 1990 at the ACM CHI'90 conference. Nielsen developed today's 10 heuristics with Molich for this very method. I cover them one by one on the Nielsen's 10 usability heuristics page. This page focuses on the method itself.

As Nielsen describes it, the output is a list of the usability problems in the interface. Each problem is written down with the heuristic it violates. The team then puts that list in the order it will fix things.

A heuristic evaluation isn't the same as a UX audit. An audit uses this method at its core and adds checks for flows, accessibility, content and speed on top.

When it helps

For me the method's biggest strength is that it needs no users. You don't recruit participants or book sessions. You can start even if all you have is a prototype.

I use it most before a user test. If you clear out the obvious problems first, you can spend the test sessions on subtler ones.

It also helps before a redesign. If you redraw a screen without seeing what's broken, you carry the same problem over to the new one.

The NN/g guide draws a clear line here: heuristic evaluation doesn't replace user research. It complements it and shows where to look in your next usability test.

How many evaluators you need

Short answer: three to five people. NN/g's 2023 guide recommends that, ideally, this many people evaluate the same interface independently. The reason is simple. However experienced they are, every evaluator misses some problems.

The advice rests on Nielsen's own data. In his 1994 article “The Theory Behind Heuristic Evaluations” he gives the average across six projects. In those projects a single evaluator found on average only 35 percent of the usability problems.

The same article has a curve. It shows how the share of problems found rises as you add evaluators, based on the average of six case studies. Looking at the curve, Nielsen says that about five evaluators is reasonable.

His usual recommended range is still three to five. He writes that adding more people brings very little new information. The model that predicts how many problems several evaluators will find was published separately by Nielsen and Landauer in 1993.

Nielsen also gives a cost example. In the sample project the most cost-effective number is four evaluators, where benefits come to 62 times the cost. That figure depends on that project's assumptions. It doesn't transfer to your project as it stands.

The practical conclusion I draw from these numbers: the second evaluator is the most valuable person you can add after the first. If your budget is tight, don't hunt for five people, but don't stop at one either. Even two people change the list noticeably.

Why do several people make such a difference? Nielsen writes that experience from many different projects shows different people find different problems. What one misses, another sees.

The session, step by step

1. 1. PrepareHeuristics, scope, how to record
2. 2. Solo session1–2 hours each, two passes
3. 3. MergeSimilar findings in one row
4. 4. SeverityEveryone rates alone, take the mean
5. 5. DebriefPriorities and next step

Figure: The stages of a heuristic evaluation. The order follows the NN/g guide and Nielsen's 1994 articles.

1. Prepare

First everyone on the team reads and understands the heuristics. NN/g suggests a practice round on a simple design. The guide's example is a weather app.

Then narrow the scope. According to NN/g, an evaluation can focus on a single task, one part of the product, one user group or one device type. “Guest checkout on mobile” is a good scope.

Finally, decide where findings will be recorded. NN/g lists three options: its own workbook, a spreadsheet, or a digital whiteboard such as Miro. In a spreadsheet each row holds one observation and the related heuristic.

2. Solo session

Each evaluator inspects the interface alone. According to Nielsen, one person's session usually takes one to two hours. NN/g also recommends capping it at 1–2 hours.

Evaluators don't see each other's notes until their own work is done. Nielsen wants findings shared only after all the evaluations are finished. That keeps every list independent.

Nielsen recommends going through the interface at least twice. In NN/g's description the first pass is for getting familiar: you do the task like a user but don't evaluate anything yet. On the second pass you note every element that violates one of the heuristics.

Each note has three things: which heuristic is violated, a short description of the problem and, if you like, a suggested fix. On a whiteboard, sticking notes onto screenshots makes this easier.

Nielsen writes that an observer can join the session. The observer records the evaluator's comments and helps them operate the prototype if it gets stuck. They don't give hints unless the evaluator is clearly struggling.

A violated heuristic isn't always a problem. NN/g explains this with the hamburger menu on mobile: because the screen is narrow, that trade-off is often unavoidable. That's why I also write down what the violation costs the user.

Merging the findings

Once everyone has finished their list, the findings are gathered in one place. NN/g suggests an affinity diagram for this. Similar findings are clustered on a physical or digital board.

The team then discusses those clusters. The NN/g guide gives five questions for the discussion:

1. Where do we agree, and where do we differ?
2. Which problems hurt the overall experience the most?
3. Which problems could be most harmful to the company's goals?
4. Which ones need more data, and which should get priority in the next usability test?
5. What steps can we take on these problems in the short and long term?

When merging I follow two rules. Notes on the same screen touching the same heuristic become one finding. A vague error message that repeats on five screens is written up as one system problem.

Nielsen also recommends a debriefing at the end of the evaluation. The evaluators, the observer if there was one, and representatives of the design team attend. The meeting also covers possible redesigns and what the design does well.

An example: a checkout flow with three people

To see the method in practice, let's set up an imaginary example. An online shop wants to evaluate its guest checkout on mobile. There are three evaluators: a designer, a product manager and me.

In preparation everyone reads the heuristics and we write down the scope: “On a phone, without an account, complete an order with a coupon.” We open a shared spreadsheet for recording, but everyone writes on their own tab.

On the first pass all three of us place the order once from start to finish, without taking notes. On the second pass we stop and compare each screen with the heuristics.

The designer notes that the coupon error appears at the very top of the page and can't be seen on a scrolled-down screen. The product manager sees something else on the same screen: the “Apply” button is active even when the coupon field is empty. I note that the selected delivery address is hard to spot.

When merging we put the spreadsheets side by side. The coupon error is on two lists, the address problem only on mine. The active button by the coupon field stays a one-person note.

Then I send the merged list to all three of us. Each of us rates every finding separately. The coupon error gets 3, 4 and 3, a mean of 3.3. The address problem gets 2 and 2.

In the debrief the coupon error goes to the top of the list. The ratings differ a little, but all three count it as at least a major problem. In the tool below you can change this example's table yourself.

How to write a finding

If notes follow the same pattern, merging goes smoothly. For every finding I ask for:

• Location: flow, step and screen name.
• Problem: from the user's point of view, in one sentence.
• Heuristic: which one it touches, with its number.
• Evidence: a screenshot or a short screen recording.

“Coupon validation is broken” is a developer note. “The coupon error isn't visible” is a title everyone on the team reads the same way. The difference matters because the person rating it may not have found the problem themselves.

How to rate severity

Severity tells you which problem to spend your limited time on first. According to Nielsen's 1994 article, the rating combines three factors: frequency, impact and persistence. Nielsen adds market impact on top:

• Frequency: how common is the problem?
• Impact: how hard is it for users to get past it?
• Persistence: do users stumble once, or every time?
• Market impact: even a problem that's easy to get past can hurt how well the product is liked.

The scale runs from 0 to 4. Below are Nielsen's definitions in my own short wording.

Rating	Name	What it means
0	Not a problem	The evaluator doesn't agree that this is a usability problem at all.
1	Cosmetic	Cosmetic problem only. Needn't be fixed unless there is extra time in the project.
2	Minor problem	Fixing it gets low priority.
3	Major problem	Important to fix, so it gets high priority.
4	Catastrophe	It has to be fixed before the product is released.

Ratings aren't given during the session. According to Nielsen, during the session the evaluator's attention goes to finding new problems. So after the sessions, everyone receives the list of all the problems found.

Every problem on the list is described in detail, with a screenshot if needed. That description is essential, because each evaluator found only some of the problems themselves. Everyone rates independently of the others.

According to Nielsen, rating takes about 30 minutes per evaluator. He considers one person's ratings too unreliable. He counts the mean of three evaluators as good enough for most practical work.

Nielsen also has a note on the release decision. If a few catastrophe-level problems remain, the interface most likely shouldn't ship. If all the remaining problems are cosmetic, shipping may be reasonable.

Try it: findings table

The tool below imitates the recording part of an evaluation session. Pick the evaluator, type the screen, then mark the violated heuristic and the severity.

If a second evaluator rates the same screen and heuristic, the row merges. The table is sorted by the mean. Rows rated by one person are flagged, because Nielsen doesn't trust a single person's rating.

Findings table

Findings table

Below are sample findings from three evaluators. Adding your own needs JavaScript.

Screen	Heuristic	Ratings	Mean	Note
Checkout · Step 2	9. Help users recognize, diagnose and recover from errors	E1: 3 · E2: 4 · E3: 3 3 people	3.3 · Major	The coupon error shows at the very top of the page and is out of view on mobile.
Checkout · Step 1	1. Visibility of system status	E1: 2 · E3: 2 2 people	2.0 · Minor	The selected delivery address is marked only by a thin grey border.
Cart	4. Consistency and standards	E2: 1 One person	1.0 · Cosmetic	The delete icon differs from the one on other screens.

The sample findings come from an imaginary checkout flow, not a real client project. Pick an evaluator, type the screen, mark the violated heuristic and the severity, then add. Ratings for the same screen and heuristic merge into one row and are averaged. The severity scale is adapted from Jakob Nielsen's 0–4 scale.

“Copy to clipboard” takes the table as plain text. You can paste it into an email, an issue tracker or your meeting notes.

Limits of the method

It doesn't say why users get stuck. The evaluator predicts a problem by looking at a heuristic. Real users may never stumble there, or may stumble somewhere else entirely.

It depends on the evaluator's knowledge. Someone who has just learned the heuristics can miss a problem or blow up a small deviation. NN/g recommends the practice round so everyone on the team understands what's expected.

It scatters over a wide scope. NN/g recommends keeping the scope narrow for large or complex products. An evaluation that starts with “the whole app” ends without looking closely enough at any flow.

It can hide what works. The method is built on looking for problems; the list counts only shortcomings. That's why Nielsen suggests also discussing the design's strengths in the debrief. It lowers the risk that the team breaks something that works while fixing things.

Severity is an opinion. Averaging reduces one person's mistake but doesn't remove it. If the ratings differ widely, such as 0 and 4, discuss that finding separately in the meeting.

Breaking a heuristic on purpose needs a reason. According to NN/g, deliberately violating a heuristic should be rare. Before such a decision, back the reason with user research.

My note

I work alone on most projects. That's my weakest point with this method. Nielsen's 35 percent figure stays on my mind. As a single evaluator I don't claim to find everything.

I balance it in two ways. I do the two passes on different days, so on the second round I can look at the product from a little further away. And I give ratings in a separate round after the list is finished, as Nielsen recommends.

When I'm not sure about a finding, I mark it clearly as such. I ask the team to test that finding with real users before any big decision. If the product is bigger than one flow, I split the scope into flows and look at each separately.