Accessibility testing tools: what do axe, Lighthouse and WAVE find?

In short

An automated accessibility tool finds what breaks a rule in the code. It can’t know whether something makes sense to the user. If you know this difference, you get more out of the tool.

• axe is an open-source testing engine. It runs as code in your own test setup.
• Lighthouse can be started from Chrome DevTools and gives the page an accessibility score.
• WAVE draws the problems over the page with icons.

All three are a start. I describe how I do the actual review in the guide to accessibility audits. This page is only about the tools.

What an automated testing tool can and can’t do

The tool reads the page’s code and the structure the browser builds from it. Then it checks each element against a list of rules. If an image is missing its alt, a button’s name is empty or the ratio between text and background is too low, it reports it.

The W3C’s Web Accessibility Initiative (WAI) draws the line clearly. According to its guide to selecting tools, tools can’t determine accessibility; they can only assist. The same guide says that tools sometimes produce false or misleading results.

The WAI’s evaluation overview says the same: no tool alone can determine whether a site meets accessibility standards. That takes the judgement of a knowledgeable person.

The rules are tied to WCAG success criteria. But not every criterion can be read from the code. The answer to “Does this alt text do the image’s job?” isn’t in the code.

axe: the rules engine

axe-core is an open-source engine for accessibility testing of web pages. According to its README, it is actively supported by Deque Systems. It contains rules for WCAG 2.0, 2.1 and 2.2 at levels A, AA and AAA, plus best-practice rules.

One of axe’s principles is not to raise false alarms. According to the README, the goal is zero false positives, bugs notwithstanding. Elements axe isn’t sure about are returned as “incomplete”, with a note that they need manual review.

That’s why, when I read an axe report, I look at two lists. The violations are tasks to fix. The “needs review” list is the order of my own checks.

A setting you should know about

According to axe’s rule descriptions, the rules for WCAG 2.2 levels A and AA are disabled by default, until WCAG 2.2 is more widely adopted and required. The target-size rule, which measures the size of tap targets, belongs to this group.

AAA rules and experimental rules don’t run by default either. If you’re aiming for WCAG 2.2 AA, turn these rules on explicitly in your test setup. Otherwise, a report that looks clean may not have looked at the 2.2 criteria at all.

The Lighthouse accessibility score

Lighthouse is an open-source, automated tool for auditing pages. It checks areas such as performance, accessibility and SEO. You can run it from Chrome DevTools, from the command line, as a Node module or through the PageSpeed Insights website.

The accessibility score is the weighted average of all accessibility audits. Each audit is pass or fail; there are no partial points. If some of the buttons on the page are missing a name, you get zero for that audit.

The weights range from 3 to 10. Lighthouse’s scoring page bases the weights on axe’s user impact assessments. Button names, the alt attribute of images and labels for form elements are among the audits with a weight of 10.

What the score doesn’t count is on the same page. Manual audits and low-impact or best-practice audits don’t affect the score. So a score of 100 says that everything the tool looked at passed. It says nothing about anything else.

If you also use Lighthouse for performance, I cover that side in the Core Web Vitals guide.

WAVE: seeing problems right on the page

WAVE is a suite of evaluation tools that WebAIM offers on its website. There are extensions for Chrome, Firefox and Edge. WAVE places the results directly over the page with icons. For designers, I find it the easiest tool to read.

According to WAVE’s help page, red icons are errors that need to be fixed. Green icons are features that probably improve accessibility, but they should be verified too. Yellow alerts mark places you should look at.

WAVE also names its own limit. It can’t tell you whether the page is accessible; only a person can determine that. And the absence of errors doesn’t mean the page is accessible or compliant.

I like one of WAVE’s habits. Because it doesn’t know whether alt text fits, it shows the text on screen and leaves the decision to you. If the photos in a product list only say “image”, you see it at a glance.

How many problems do automated tools find?

One of the figures on this question comes from axe’s README. It says that with axe-core you can find, on average, 57% of WCAG issues automatically. Before using this figure, it’s worth looking at how it was measured.

The figure is based on Deque’s own report on automated testing coverage. The report looks at more than 13,000 pages and nearly 300,000 issues from over 2,000 audits. Only first-time audits were included. Of 294,958 issues in total, 169,242 were found with automated tests, which is 57.38%.

What matters is this: the share is calculated by the number of issues, not the number of criteria. The same report says that automated issues were found for only 16 of the 50 success criteria in WCAG 2.1 AA. According to the report, some types of issues occur much more often than others. 73,733 of the 169,242 automatically found issues come from a single criterion, 1.4.3 Contrast.

The report has other limits. It covered only HTML pages and levels A and AA of WCAG 2.0 and 2.1. The data comes from audits Deque carried out with its own tools and its own methodology. So the company doing the measuring is the same one selling the tool.

Here’s how I read the figure: the tool clears away a large share of the common errors quickly. But for most criteria, a person has the final say.

Try it: what does the tool find?

The eight cards below are made-up examples of problems I often see in audits. For each card, choose: does an automated tool find this, or does a person have to look by hand? After you choose, the answer and the reasoning open below the card.

What does the tool find?

• Pale grey help text

  A small, light grey hint text sits on a white background. The contrast ratio is below 4.5:1.

  Tool finds it. The tool measures the ratio of text on a solid background. That’s exactly what axe’s color-contrast rule looks for. The same rule sometimes returns “needs review”, and then you measure it yourself.

• Product photo without alt text

  The main photo on the product page has no alt attribute at all on its img element.

  Tool finds it. That the attribute is missing is right there in the code. axe’s image-alt rule finds it, and so does one of Lighthouse’s most heavily weighted audits.

• Meaningless alt text

  The image has alt="IMG_2041.jpg". The attribute is there, but it doesn’t say what the image shows.

  Only by hand. The tool sees that there is alt text and lets it pass. Whether the text replaces the image is for a person to decide. That’s why WAVE shows the alt text on screen and leaves the decision to you.

• Icon button without a name

  The trash can icon in the cart is a button. Inside it there is only an SVG: no text and no aria-label.

  Tool finds it. The button’s accessible name is empty. axe’s button-name rule finds this, and Lighthouse counts the same audit at the highest weight.

• Email field without a label

  The newsletter form has an email field. No label or other name is associated with the field.

  Tool finds it. The association with the label is in the code, and the tool reads it. axe’s label rule requires a label for every form element. Whether the label is understandable, though, is again for you to check.

• Jumbled focus order

  In checkout, the Tab key jumps from the name field to the links at the bottom of the page and then back to the card number.

  Only by hand. Because each element can be focused on its own, no rule violation shows up. You only notice whether the order is logical when you go through it with Tab.

• Keyboard trap

  A promo window opens. The Tab key cycles inside the window, Esc doesn’t close it, and the close button can’t be reached.

  Only by hand. When it reads the code, the browser doesn’t try to open and close the window. The trap is found by someone who tries to get in and back out with the keyboard.

• Errors shown only in red

  The border of the invalid field turns red. There is no text or icon next to it.

  Only by hand. Everything is there in the code: the field and the label. That the error is conveyed only by colour is something only a person looking at the screen can see.

The cards are made up. The grouping is mine: what I call “Tool finds it” has a match in axe’s rule list; what I call “Only by hand” doesn’t show up as a rule violation in the code. On a real page, you still confirm the tool’s result with your own eyes.

The four problems you only find by hand have one thing in common: the code looks “right”. There is a label, there is alt text, the elements can be focused. The problem lies in what all of it together tells the user.

Checks no tool replaces

When the tool report is clean, I do this short round:

1. I put the mouse aside and try to complete the main flow with the keyboard only. The focus ring must be visible at every step, and the order must follow the reading order on screen.
2. I go into every window that opens and try to get back out with Esc and Tab. When the window closes, focus must return to the button that opened it.
3. I measure the contrast of text on photos or gradients by hand. Enter the colours into the contrast checker to see the ratio.
4. I fill in the form wrong on purpose. The error must be conveyed not only by colour but in text too. I show how in the form design pattern.
5. I turn on the screen reader and listen to the images’ alt text and the buttons’ names.

This round doesn’t find everything, but it looks at most of the places where the tool is blind. You’ll find the steps of a full review in the audit guide.

I don’t cover legal obligations on this page. For the EU, read the guide to the European Accessibility Act.

My note

When teams come to me saying “We have 100 in Lighthouse, so we’re accessible”, I first ask them to pay with the keyboard. We usually get stuck at the first window that opens. The score isn’t wrong. It just never asked that question.

On choosing a tool, I have only one piece of advice: how often you run it matters more than which tool you pick. If you build axe into your test setup, the same errors don’t come back with every release. I give WAVE to design and use Lighthouse for a quick first look.

I also don’t just copy findings from the tool report. In my report, I only include problems I have confirmed on screen with my own eyes. If you want this done for your product, the UX audit includes the core WCAG 2.2 checks.