Task success rate: what it measures and how to calculate it

What the task success rate measures

This metric answers a single question: could the person get the task done? Jakob Nielsen and Raluca Budiu describe it in the NN/g article as the simplest usability metric. It is the share of participants who complete the task, and it is measured in binary terms.

The formula is: success rate = successful participants / all participants × 100. Imagine you give 12 people the task “Send $500 to a saved recipient” in a banking app. If nine send the money to the right person, the rate is 75%.

Before measuring, write down what counts as success. A clear end point such as “The confirmation page was opened” prevents arguments during analysis. How to word the task is in the usability testing guide.

What is a good rate? Jeff Sauro at MeasuringU examined 1,189 tasks from 115 tests. 3,472 users took part in these tasks, and the average completion rate was 78%. I see this number as a starting point for comparison.

Binary success and partial success

Calculated in binary terms, a person either succeeds or fails. This strictness sometimes seems unfair. Has someone failed who sent the money but left the memo blank?

NN/g therefore writes that it sometimes gives partial credit. In the example of a flower order there are four levels: full success, success with a minor problem, success with a major problem and failure. A minor problem is a missing gift card, for example. A major problem is a wrong delivery date or a wrong address.

The same article also warns. It is wrong to give these levels numbers such as 1, 0.66, 0.33 and 0 and take the average. The numbers are only ordered labels. NN/g reports each level as a rate of its own with its own confidence interval.

Most teams still want a single number. That is why I built the “count partial success as half” option into the calculator. With both numbers side by side, it is easier to discuss where the difference comes from.

Whatever you decide: settle the criterion for partial success before the test. A sentence such as “Memo blank = partial success” is enough. If the criterion is only written after the session, two people rate the same recording differently, and the number becomes a point of contention.

Calculator: success rate and time on task

Task success rate calculator

Results of one task

Participants

Successful

Partial

Failed

How should partial successes count?

As failedBinary: only full success counts. Count as half (0.5)Each partial success counts as 0.5 success.

Success rate

60%

Full success, 95% interval

31% to 83%

6 / 10 = 60%. Partial successes count as failed.

Task times

Times (seconds, separated by commas)

Use a point for decimals, such as 12.5. The comma separates two times.

Number of times

8

Mean

51.9 s

Median

42.5 s

Geometric mean

47.7 s

Because JavaScript is off, the results only apply to these example values. The formulas are on the page.

The tool stores and sends no data. The calculation runs in your browser.

Enter the numbers for one task. The success rate and time values are recalculated as you type. The confidence interval refers only to full successes.

Checking the calculation by hand

Before I trust a tool’s result, I work through a few examples by hand. You can enter the three examples below into the calculator and compare the results.

Input	Success rate	Times (s)	Mean / median
10 people: 6 successful, 2 partial, 2 failed	Binary 60%, counted as half (6 + 1) / 10 = 70%	34, 41, 38, 52, 47, 39, 120, 44	415 / 8 = 51.9; in the sorted list, the middle between 41 and 44: 42.5
5 people: 4 successful, 0 partial, 1 failed	80%, interval roughly 36% to 98%	60, 30, 45	135 / 3 = 45; middle value 45
20 people: 10 successful, 6 partial, 4 failed	Binary 50%, counted as half (10 + 3) / 20 = 65%	10, 20, 30, 40, 1000	1100 / 5 = 220; middle value 30

Look at the times in the first row. One of eight people needed 120 seconds. That single value pulls the mean up to 51.9 seconds. The median stays at 42.5, the geometric mean at 47.7 seconds.

The third row is an extreme example of skew. A single session of 1,000 seconds raises the mean to 220. Yet four of five people completed the task in 40 seconds or less.

Time on task: mean or median?

Time on task is the time a person needs to complete the task. It is easy to measure. A common mistake in reporting, though, is giving only the arithmetic mean.

According to Sauro, task times are right-skewed. Some users get stuck in the interface or work more slowly. A few long times pull the mean up, and the mean no longer shows the typical time.

Sauro’s example shows this well. The mean of 100, 101, 102, 103 and 104 seconds is 102. Add 200 seconds, and the mean rises to 118.33, the median only to 102.5.

Sauro’s recommendation depends on sample size:

• Under 25 users, use the geometric mean. According to Sauro, it estimates the middle time most accurately in small samples. The median can overestimate the true value here by up to 10%.
• Over 25 users, the median is a good choice.

For the geometric mean, take the logarithm of each time, average those, and convert the result back. You don’t have to do that by hand. The calculator above shows mean, median and geometric mean side by side.

What about the times of those who failed?

Sauro advises not deleting the times of those who fail the task. On reviewing the recording, a failure can turn out to be a success. With these times you can report three measures: completion time of the successful participants only, time to failure and the total time everyone spent on the task.

When success is low, this distinction matters. In Sauro’s example, only 4 of 12 people complete the task. Reporting only those four times can make efficiency look better than it is.

Error rate

An error is a wrong step on the way to the goal. Sauro distinguishes two kinds. A slip is an unintended action with the right goal, such as a typo. With a mistake, the goal itself is wrong. Even if the person completes the task with that wrong goal, it is an error.

The error rate can be calculated in two ways:

• Binary: anyone who makes at least one error in the task gets a 1; anyone who makes none gets a 0. You lose information, but for tasks with few errors that can be enough.
• By error opportunities: divide all errors by all error opportunities. If the task has 5 error opportunities and 10 people attempt it, that makes 50 opportunities. With 5 errors, the rate is 5 / 50 = 10%.

The hard part is defining the error opportunities in advance. In forms, I count each field as one opportunity. That way you also see in form design which field causes problems.

Confidence intervals with small samples

If four of five people complete the task, your product’s success rate isn’t 80%. 80% is only the result of those five people. Where the true rate might lie is what the confidence interval tells you.

Sauro recommends the adjusted Wald method for completion rates. According to him, it gives the best coverage for samples under about 150. The formula adds z²/2 to the successes and z² to the participants; for 95% confidence, z = 1.96.

I calculated 4 of 5 myself with this formula. The 95% confidence interval lies roughly between 36% and 98%. So the true rate could also be under half.

In his 2006 NN/g article, Nielsen recommends 20 users for quantitative studies. At 90% confidence, the margin of error with 10 users is about ±27% of the mean, with 19 users ±19%. How many people which study needs is covered in detail in the guide how many users to test.

The NN/g article on success rate reaches the same conclusion. Most insights come from tests with 4–5 users thinking aloud. But these tests don’t give the narrow confidence interval that metrics need.

Reading the metrics together

Nielsen’s 2001 article on usability metrics names four measures: success rate, time on task, error rate and subjective satisfaction. None of them shows the whole picture on its own.

If the success rate is high but the time long, the person completes the task only with effort. If the time is short but there are many errors, the screen may be pushing people to rush. If you want a standard questionnaire for satisfaction, use the SUS calculator.

In the same article, Nielsen uses the geometric mean when he combines improvements per task into one value. The reason: a single large number shouldn’t distort the result. He also writes that adding up the times is simpler but can mislead if the tasks don’t occur equally often.

My note

The reporting mistake I see most often in audits: the result of a test with five people goes to management as “80% success”. The number looks exact, and nobody asks about the interval. When the new design scores 60% six months later, the team panics, although both results lie in the same interval.

My rule is simple. With small tests, I use numbers only as a direction and always add the interval. If I want to compare, I go to 20 people. I count success in binary terms and show partial successes on a line of their own.

I do the same with time. I never write the mean alone; I add the median or geometric mean and how many people failed. A single long session is enough to distort the mean.