What 113 accessibility professionals say about auditing
Only one in nine auditors work through the WCAG list and nothing else. Here’s how the rest do it, and why the order matters for the report.
I asked 113 accessibility professionals whether they audit criteria-first or issues-first. Most do both, for four different reasons: scope, speed, real barriers, and the issues WCAG doesn’t cover. What they said, how I work, and what it means for the tools we build.
I recently posted a poll on LinkedIn called How do you approach manual accessibility testing? I asked: "When you carry out an audit, do you work through a list of WCAG criteria to check against, or find issues and then match them with WCAG criteria? Or a mix of both, or neither?"
What people said
When the poll closed, 113 people had voted. 46% use a mix of both, 39% find issues first and map them to criteria afterwards, 11% start from the criteria and 2% do neither (I have no idea what that 2% do but I would love to know).
Essentially, what I got from the votes and the comments is that the checklist stops you missing things. On a formal conformance assessment you walk the criteria so that nothing gets missed, although one reply pointed out that even then criteria can be grouped, so with headings checked under 1.3.1 and 2.4.6 in the same pass. Another described the mix in one sentence: start from the list, follow up other failures as they turn up, then go back to the list to make sure everything’s been covered.
Going criterion by criterion is slow, though. One reply set out a test order by activity instead: code inspection, screen reader, keyboard, contrast, then reflow, resize and text spacing, with the automated scan last. That's a really interesting approach to me because I'll always start with a scan to quickly see the shape of things. In fact I won't quote a piece of work before I've run a scan because the amount of time required to complete an audit is largely defined by how good the accessibility is, and a scan - while not giving you the whole picture, definitely gives you a good thumbprint.
Scope changes things too. Even for an informal review, one comment said that it makes sense to check the common failures first. If a product has no text alternatives or keyboard support, a full assessment wastes everyone’s time, because it will all need retesting. That makes a lot of sense to me.
Some auditors start with people. One team puts disabled testers through the main journeys alongside an automated scan, reports any blocker straight away and maps to criteria later. Someone else runs AI first, then keyboard and screen reader testing. That’s a different discussion, and I’ll come back to it another time.
Three people in the comments made the same point: not every issue is a WCAG issue. The clearest example came from a professional working in e-learning. In the Rise authoring tool, an image with zoom turned on and alt text passes 1.1.1, but a screen reader announces zoom, image, zoom, image before it gets to the alt text. Nothing fails. It’s just noise, and the zoom does nothing the browser can’t already do. Another turns on the scanner’s best-practice rules for the same reason: content outside a landmark maps to no criterion and is still worth reporting.
What the mix looks like for me
To share what I normally do, pretty much every audit starts with a scan in WAVE or Silktide. The number and type of failures tell you what you’re likely to find, and they prepare you for one of two outcomes: "it’s not too bad" or "this is going to be a nightmare". It's essential because I remember starting a test with a scan of a particular retail site. Not a single image had alt text. At that point you know there’s no accessibility focus anywhere in the organisation and you’re starting from scratch.
You’ll be writing up a large volume of issues, and you’ll have to go to greater lengths to explain why each one is a problem because you know that the knowledge level (with regards to accessibility) is close to zero. I’m happy to say that it’s increasingly rare that this happens, but it happens. And this is why I always run a scan before quoting work; because the number of issues you find is what defines the time you spend on an audit, not the number of screens.
The type of product also has some impact on how an audit is conducted. With a SaaS product (say a project management or HR platform) I play with it. I want to know what it feels like with a keyboard and a screen reader, whether I can reach all the different parts easily, and whether the landmarks are labelled sensibly. Even where the accessibility fundamentals have been considered, I assume usability for disabled people probably hasn’t been thought about entirely. If I was testing a simple news site then I wouldn't be as concerned with things like landmarks which will allow the user to jump to different sections.
I also make assumptions depending on the sector. A lot of my work has been for UK government agencies, whose services are often a step-by-step process. When a team follows the GOV.UK guidance, the service is usually exceptional: easy to test, with a short list of issues to raise.
Scan results usually come out in WCAG order and go into the report that way, but every one of them has to be checked by hand (always, always, always). While checking them you'll find other things, and those go into the report immediately. You might not come across them again. This is the part where WCAG order doesn’t apply, whenever you find an issue, you should write it up. It's possible that you might miss it when you test again.
Only then do I work through the criteria, page by page. If I’ve already logged something under 1.1.1, I still check the rest of the page against it. Things that have been flagged by automated tests or that I happen to come across when I've been looking at something else are unlikely to uncover everything that you might find. so doing a methodical check for things criterion-by-criterion is important.
I’ve often thought about a faster order that groups the criteria by test activity: a screen reader pass, a keyboard pass, changing styles and resizing the window, forms, labels and errors, image alt text, and the automated results. It’s close to the activity-based order mentioned in the comments of the poll, and to the habit of checking similar issues like 1.3.1 and 2.4.6 together. The WCAG sweep would still come at the end. I’m going to turn that into a shareable testing methodology, using what people described in the comments, and I plan to publish it here when it's ready.
What about retests?
There are a lot of variables here, with the main one being the volume and frequency of issues found in the first test. But having said that, retests are narrower. I usually stick to the pages in the original test, depending on budget. Anything I come across on other pages still gets recorded, with a note that it was out of scope for the original test, which recognises that it wasn't something that was missed in the original scope, it's something found outside of it.
One defect, several criteria
When a single control fails more than one criterion, I write it up against each criterion it fails, with the reason it fails that one. The same screenshot often does for all of them. (Every issue needs evidence: a screenshot, a video, or the code where a screenshot doesn’t make sense). I'll do them all at the same time while they are fresh in my mind. It's important to bear in mind that cross-referencing the issues helps designers or engineers identify that several issues relate to one component.
Where this leaves tools
Most audit tools are built criteria-first, because that’s the shape of the standard: a numbered list of success criteria, each one passed or failed. One in nine. That’s how many of the people who answered work the way most audit tools are built, including ours. And to help speed up the testing process using Labrador and align it with what people told us in the poll, we’re looking into adding some features that will do that and will release them in the next few weeks.
Over to you
Thank you to everyone who voted, and especially to the people who explained in the comments how they work. Most of this piece comes from what you wrote.
One thing the poll didn’t ask: does your approach change with the client, or with the tool you’re using? If it does, tell me how in the comments on the LinkedIn post. And if you’d like to see the testing methodology when it’s out, follow me on LinkedIn.
Further reading
You might also be interested in some of these articles by my co-founder Sam Donnelly: How long does an accessibility audit take? explains why audit time depends on the templates, flows and number of issues more than the page count, and Manual vs. automated accessibility testing covers what a scan catches and what still needs a person testing with assistive technology.