Automated testing for accessibility

We took a closer look at three of these tools: the axe DevTools - Web Accessibility Testing Plug-In for Google Chrome, WAVE from WebAIM and the Free Accessibility Checker from Silktide.

axe DevTools is a browser extension for Google Chrome and therefore only works in this browser. After installation, the tool can be opened via Chrome's developer tools. The analysis is based on WCAG 2.1 and was tested in the free version.

Operation is straightforward: The tool can be used directly on any page to be analyzed. It clearly highlights errors by marking the respective elements and describing the associated problems, and also links to the relevant WCAG guidelines. This makes the tool easy to understand, even for non-developers. However, there is a point deduction here for clarity - the many code snippets, tags and further information can quickly overwhelm users and overwhelm people who are not familiar with developer tools.

The test shows that the tool has certain limitations. For example, it does check the contrast of UI elements such as buttons, but only in the default state - hover or pressed states are not taken into account. Strictly speaking, this is not WCAG-relevant, but a manual test option would still be practical here in order to create an inclusive user experience regardless of guidelines. Another problem occurs with an element that places white text on a bright image. Although this text is difficult to read even for people with full visual acuity, the tool does not recognize the contrast error. In general, it is not clear what the tool checks and what needs to be checked manually.

The second tool in the test is WAVE from the provider WebAIM, an organization that has been committed to web accessibility since 1997 and is particularly well known in the USA. WAVE is available as a browser extension for Chrome, Firefox and Microsoft Edge. Alternatively, the tool can also be used directly via the website by entering the URL of the desired page. The analyses are based on the current WCAG 2.2 guidelines.

One advantage of WAVE is that it highlights errors directly on the page and also provides links to the associated code. Another positive feature is the in-depth checking of alternative texts: The tool not only points out missing alt texts, but also issues warnings for particularly long texts. This function supplements the check with additional useful parameters. WAVE also displays all existing alt texts, structural elements and ARIA labels, but this can quickly become confusing for inexperienced users as there is no clear visual distinction between incorrect and correct elements.

The integrated Contrast Checker is an exciting feature that can be used to display the page in grayscale and check the contrast of individual elements. However, the tool points out that gradients, transparencies and filters in the background cannot be taken into account. Background images must be tested manually using an eyedropper.

A major criticism of WAVE is the interface: It seems confusing and the operation is not very intuitive, especially when it comes to calling up reference texts or links to the relevant WCAG criteria for error messages. Despite these weaknesses, WAVE provides a wealth of information and is therefore particularly suitable for experienced users. For a first impression of the accessibility of a website, the tool is a solid choice as it tests according to the latest WCAG guidelines.

The Free Accessibility Checker from Silktide is also available as a browser extension, but again exclusively for Google Chrome. In addition to the free extension, Silktide also offers comprehensive accessibility software, but our test relates exclusively to the free version of the checker.

An outstanding feature of this tool is the option to select the WCAG version and conformance level (A, AA or AAA) to be tested before the analysis. Similar to WAVE, the checker performs an automated test at the touch of a button and then displays error messages that highlight the corresponding element on the website when clicked. Particularly practical: each error message not only contains the appropriate WCAG criterion, but also a detailed explanation, suggested solutions with code examples and further links.

The tool's interface is clearly laid out and intuitive to use. Silktide also offers small instructions or videos in many places to make operation easier - a big plus point for user-friendliness.

Particularly impressive are the additional features that no other tested tool offers to this extent. The Checker not only enables the simulation of screen reader output and the operating sequence via the keyboard, but also the simulation of various visual limitations. These include double vision, cataracts and loss of peripheral vision.

Silktide also offers an extraordinary variety of color perception: in addition to the classic grayscale view, different color vision deficiencies can be simulated. The simulation of dyslexia is also impressive, even if it is unclear to what extent this reflects reality. In addition, all simulations are supplemented with explanatory texts and statistical data on the frequency of the impairments.

Silktide sets new standards with these functions and offers both a precise technical analysis and deeper insights into the user's perspective. Thanks to its extensive features, user-friendly interface and detailed help, Silktide's Free Accessibility Checker is currently our clear favorite among the tools tested.

Accessibility checkers are undoubtedly helpful tools to quickly identify basic accessibility issues on a website. They can help to uncover obvious errors and check technical standards of some WCAG criteria. However, as we have found, these tools quickly reach their limits - both technically and in terms of content.

A key weakness is their inability to detect contextual issues. A checker may confirm that all alternative texts are present, but if they only consist of the word “image”, this is completely useless for users. Similarly, many tools fail to check complex, interactive elements as they are often unable to analyze them correctly. There is also a certain amount of room for interpretation with some WCAG guidelines. While the checkers can only test clearly measurable criteria, some guidelines require context-dependent consideration.

No tool can reliably evaluate the user experience either. Aspects such as visual hierarchies, real usage contexts for contrasts or intuitive navigation require human judgment. The same applies to checking keyboard operation or screen reader compatibility. Although some tools, such as Silktide's Free Accessibility Checker, offer helpful screen reader simulations, they cannot assess whether the sequence, structure and content of what is being read out makes sense.

These tools can be used to detect technical and easily identifiable problems and help to gain an initial overview. Nevertheless, manual testing by real users is essential if the accessibility of a website is to be not only technically correct, but also truly inclusive and user-friendly.