Automated vs. Manual Accessibility Audits
Accessibility is not only about coding; it is also about making sure that the code serves people with disabilities as intended. How can one make sure? In the world of ADA compliance and WCAG conformity, there are two main approaches to website accessibility testing: those who rely on automated tools and those who test everything manually.
Actually, none of these methods alone is enough. While automated testing leaves huge gaps in user experience, manual testing is slow and inaccurate. In this article, we'll discuss both automated and manual accessibility testing, how to perform it, what is good and bad about it.
Automated Accessibility Testing: A First Line of Defense
Automated accessibility testing consists of scanning your website's HTML, CSS, and ARIA markup with special software, browser extension, or command-line tool according to some set of WCAG rules. The tool analyses your code and detects objective problems.
How to Perform Automated Testing
Starting automated testing is simple. Here is a standard workflow:
Choose a Tool: Some of popular tools include Axe DevTools by Deque, WAVE by WebAIM, and Google Chrome's built-in Lighthouse. You also have our free single page scanner and our full site scanner.
Run the Scan: On our scanner, you enter the URL or domain you want to scan, and click "Scan".
Review the Report: The tool will create the report with detected issues instantly. The most common errors include missing alt texts on images, empty links, lack of color contrast, missing form labels. With our scanner, you can export your report as JSON, CSV, or PDF with a free account.
The Benefits of Automation
Speed and Scalability: Automated tools are capable to check hundreds or thousands pages within several minutes. It would take weeks to do that manually.
Catch "Low-Hanging Fruit": They are perfect for detecting objective syntactic mistakes (missing IDs, empty attributes) which the developers sometimes forget about.
No Accessibility Expertise Required: Any developer or QA tester can launch Lighthouse and correct detected HTML problems even if he or she knows nothing about assistive technology.
The Limitations of Automation (The 30% Rule)
The most dangerous pitfall in accessibility is the fallacy of "Zero Error". It means that absence of any violations in the report does not guarantee that your site is accessible. On the contrary, industry experts agree that automated tools can detect only 30% to 40% of violations.
An automated tool can tell let you know if an image has an 'alt' attribute or not, but it cannot let you know if the alt attribute makes sense to a human. Automation cannot judge the context, usability, or logic of the page. It cannot tell you if the keyboard navigation is confusing, or custom widget is inaccessible. For such cases you need a human.
Manual Accessibility Testing: A Human Factor
Manual accessibility testing consists of a person, usually expert in accessibility, and sometimes a real user with disabilities, navigating the website using the same assistive technologies and techniques that real users use.
How to Perform Manual Testing
Manual testing is very subjective and based on personal experience. A standard manual audit consists of:
Keyboard-Only Navigation: You should unplug your mouse and try to reach every link, button, and form using Tab, Enter, Spacebar, and Arrow keys. Every focused element should have visible focus ring. Also you should check that tab order is logical.
Screen Reader Testing: You should launch the screen reader (VoiceOver for Mac, NVDA or JAWS for Windows), close your eyes or turn off your monitor, and try to navigate the page. Buttons should announce their purposes, all the error messages should be read aloud dynamically.
Zooming and Scaling: You should scale the page by 200%, 400%, and more. Does the layout break? Is text readable? The site should not require two-dimensional scrolling (horizontal and vertical) to read one paragraph.
Cognitive Walkthroughs: Checking for overly complicated language, flashing animations without pause ability, and lack of instruction in forms.
The Benefits of Manual Testing
It allows to detect contextual mistakes: a human can decide if the headings structure makes logical sense, if
aria-labelcorrectly describe the action of button.Validate actual usability: a site may be conform to WCAG, but still extremely difficult to use with screen reader. Manual testing is concentrated on the experience of users, not code.
Test complex interactions: modal dialogs, dropdown menus, drag-and-drop interfaces, state changes in single-page applications almost always require manual testing to make sure that everything works well with assistive technology.
The Limitations of Manual Testing
Very time-consuming and expensive: auditing whole enterprise website manually takes a lot of time and money.
Requires special knowledge: using a screen reader effectively takes some practice. An inexperienced tester can log false positive errors just because he or she does not know how to use this tool correctly.
Prone to human error: it is easy for a human tester to miss a broken link or a subtle color contrast problem hidden somewhere deep on the page. Machine would notice it instantly.
Conclusion: Automated and Manual Testing Must Work Together
In the pursuit of ADA compliance, competing automated vs. manual testing is a false dilemma. These are not rivals, but two pieces of a whole picture of accessibility strategy.
The best practice workflow is a hybrid one: first of all, you can use automated tools during development process to instantly detect syntax and obvious violations. And after the automated tests pass successfully, you can involve human experts to test keyboard navigation, screen reader navigation, and general usability of the interface. If you need manual testing or remediation, contact us today.