Automated checks for accessibility rules and standards

[saw-jan]

Automated Checks for Accessibility Rules

Accessibility issues like these (but not all) could have an automated checks:

• [a11y] 9.4.1.2 - Name, role, value available #5397
• [a11y] 9.1.3.1e - Data tables implemented correctly #5380
• [a11y] 9.1.3.1a HTML structural elements for headlines #10722
• + accessibility standard rules violations

Why automate?

• Checking a11y rules is easy and quicker than manual
• Checks can be brought to the CI pipeline
• Early accessibility assessment

Why not depend on automated tests ONLY?

• Automated tests can only check if certain rules have been applied to certain elements or not
• Cannot guarantee the accessibility of a complete feature

Available Tools

Manual

• Google Lighthouse (available on chrome browser)
• Accessibility Insights
• ...

Automation

• axe-core (Most tools and libraries use this under the hood)
• e2e @axe-core/playwright
• component vue-axe@next

POC

• Via e2e Test - @axe-core/playwright
• Via Component Test - axe-core

Reports

Lighthouse	Accessibility Insights

Automated (axe-core):

{
  "description": "Ensure buttons have discernible text",
  "help": "Buttons must have discernible text",
  "helpUrl": "https://dequeuniversity.com/rules/axe/4.10/button-name?application=playwright",
  "id": "button-name",
  "impact": "critical",
  "failureSummary": "Fix any of the following:
    Element does not have inner text that is visible to screen readers
    aria-label attribute does not exist or is empty
    aria-labelledby attribute does not exist, references elements that do not exist or references elements that are empty
    Element has no title attribute
    Element does not have an implicit (wrapped) <label>
    Element does not have an explicit <label>
    Element's default semantics were not overridden with role=\"none\" or role=\"presentation\"",
  "html": "<button data-v-babade49=\"\" type=\"button\" class=\"oc-button oc-rounded oc-button-m oc-button-justify-content-center oc-button-gap-m oc-button-passive oc-button-passive-raw oc-ml-xs\" id=\"users-filter-confirm\">",
  "none": [],
  "target": ["#users-filter-confirm"],
  "tags": [
    "cat.name-role-value",
    "wcag2a",
    "wcag412",
    "section508",
    "section508.22.a",
    "TTv5",
    "TT6.a",
    "EN-301-549",
    "EN-9.4.1.2",
    "ACT"
  ]
}

Proposal

Add automated accessibility checks:

• Rules checks - in ONE of the following forms
  • Component level
  • Separate e2e (Playwright)
  • Integrate with existing e2e (Playwright - Cucumber)
• Keyboard only navigation - e2e (?)

[saw-jan]

^ @ScharfViktor @kulmann @JammingBen @kobergj

[saw-jan]

• Rules checks - in ONE of the following forms
  • Component level
  • Separate e2e (Playwright)
  • Integrate with existing e2e (Playwright - Cucumber)

@LukasHirt Could you recommend/suggest at which level we should have the automated accessibility checks?
e2e would be more closer one to the end user experience. Also, if we use the existing playwright-cucumber setup (checking page accessibility using gherkin), it would be very quick to get started.

[LukasHirt]

Could you recommend/suggest at which level we should have the automated accessibility checks?

How would it look like if it would be implemented as part of e2e tests? Would it go through every page one-by-one? Would that mean we need to handle setup of different scenarios to cover all the screens?

Generally speaking I am a bit hesitant to have this as part of e2e tests. Having to spin up the whole setup to check a11y roles, etc. feels like an overkill. Since this feels more like a "linter" (please correct me if I am wrong here) it looks more to me like a test that should be run outside of playwright.

Could you give me please some better understanding what's the difference in effort between these two approaches? I guess if you say that having this as part of e2e tests would be 10x faster to develop as POC than having it at the component level, I would say go for it. If the difference wouldn't be that drastic I would personally prefer trying the component level first. But that's my 2 cents. If you still believe that playwright for this is the way to go, I'm not going to block you.

[saw-jan]

How would it look like if it would be implemented as part of e2e tests? Would it go through every page one-by-one? Would that mean we need to handle setup of different scenarios to cover all the screens?

yeah exactly. e2e means accessing web via url, navigating to the page and do the accessibility checks. that means it tests the accessibility of integrated components.

As far as I can tell, with the component level, effort wise, we probably need to add a lot tests as there are large number of components and there will involve mocking and tests with state changes. This can do the element-level isolated checks.

But with e2e and since we already have working setup for e2e, we just have to implement one or two test steps and write an accessibility feature then run the test using the existing e2e test run command. Also, checking the whole page or certain part of the page would be closer to how/what the end users will experience.
And if we integrate to existing e2e tests, maybe we can also have some scenarios or journey for keyboard only navigations later on.

I would say, it would be easier to get started with e2e test because of the existing setup. But with the component level, we might get lost in the large volume of available components in different packages. 😅

[LukasHirt]

As far as I can tell, with the component level, effort wise, we probably need to add a lot tests as there are large number of components and there will involve mocking and tests with state changes. This can do the element-level isolated checks.

I see. I misunderstood than how this would work. I was imagining something alongside of a scan of codebase and just checking whether the roles are applied in the code. Then yes, POC with e2e tests seems more reasonable to me as well.

[saw-jan]

I was imagining something alongside of a scan of codebase and just checking whether the roles are applied in the code

This is also valid but would be good to start with rendered html.
Thanks Lukas, we will add checks for some important pages first.

[koebel]

We have implemented an accessibility test for the basic functionalities for file view (personal space, without sidebar) as an example/proof of concept for what aspects of accessibility can be tested in an automatic fashion using the axe-core plugin.

Motivation for this is making sure that oCIS web is compliant with the WCAG accessibility standards in light of European Accessibility Act (EAA) being implemented in June 2025

Next steps

• decide which accessibility guidelines should be included (currently we scan for WACA & best practice violations, see list of all available tags offered by axe-core)
• decide where to report findings and processes to make sure that the issues will get fixed
• consider if/which of the following aspects should be covered with similar accessibility tests
  • login page
  • app header functionalities (search, notifications, account page, app switcher, etc)
  • left navigation sidebar of files view
  • side bar with sub menus of files view
  • shares (only functionalities that differ from personal files view)
  • spaces? (it doesn’t seem to have any unique UI elements that aren’t used in other views)
  • deleted files (trashbin)
• refactor page object if a lot more tests get implemented, create accessibility utility function

preliminary findings of usability issues
(TBD where to document them, see open point above)
some of them where detected through the tests implemented in #12155, others (issues about login) through further spot checks (proper test scenario not yet implemented)

issues with table view

``` { id: 'button-name', impact: 'critical', tags: [ 'cat.name-role-value', 'wcag2a', 'wcag412', 'section508', 'section508.22.a', 'TTv5', 'TT6.a', 'EN-301-549', 'EN-9.4.1.2', 'ACT' ], description: 'Ensure buttons have discernible text', help: 'Buttons must have discernible text', helpUrl: 'https://dequeuniversity.com/rules/axe/4.10/button-name?application=playwright', nodes: [ { any: [Array], all: [], none: [], impact: 'critical', html: '', target: [Array], failureSummary: 'Fix any of the following:\n' + ' Element does not have inner text that is visible to screen readers\n' + ' aria-label attribute does not exist or is empty\n' + ' aria-labelledby attribute does not exist, references elements that do not exist or references elements that are empty\n' + ' Element has no title attribute\n' + ' Element does not have an implicit (wrapped) \n' + ' Element does not have an explicit \n' + ` Element's default semantics were not overridden with role="none" or role="presentation"` }, { any: [Array], all: [], none: [], impact: 'critical', html: '', target: [Array], failureSummary: 'Fix any of the following:\n' + ' Element does not have inner text that is visible to screen readers\n' + ' aria-label attribute does not exist or is empty\n' + ' aria-labelledby attribute does not exist, references elements that do not exist or references elements that are empty\n' + ' Element has no title attribute\n' + ' Element does not have an implicit (wrapped) \n' + ' Element does not have an explicit \n' + ` Element's default semantics were not overridden with role="none" or role="presentation"` } ] } { id: 'link-name', impact: 'serious', tags: [ 'cat.name-role-value', 'wcag2a', 'wcag244', 'wcag412', 'section508', 'section508.22.a', 'TTv5', 'TT6.a', 'EN-301-549', 'EN-9.2.4.4', 'EN-9.4.1.2', 'ACT' ], description: 'Ensure links have discernible text', ✓ 2 …ec.ts:19:5 › Accessibility › personal space page › files section should not have any automatically detectable accessibility issues (2.8s) helpUrl: 'https://dequeuniversity.com/rules/axe/4.10/link-name?application=playwright', nodes: [ { any: [Array], all: [], none: [Array], impact: 'serious', html: '', target: [Array], failureSummary: 'Fix all of the following:\n' + ' Element is in tab order and does not have accessible text\n' + '\n' + 'Fix any of the following:\n' + ' Element does not have text that is visible to screen readers\n' + ' aria-label attribute does not exist or is empty\n' + ' aria-labelledby attribute does not exist, references elements that do not exist or references elements that are empty\n' + ' Element has no title attribute' } ] } ```

issues with login

``` { id: 'autocomplete-valid', impact: 'serious', tags: [ 'cat.forms', 'wcag21aa', 'wcag135', 'EN-301-549', 'EN-9.1.3.5', 'ACT' ], description: 'Ensure the autocomplete attribute is correct and suitable for the form field', help: 'autocomplete attribute must be used correctly', helpUrl: 'https://dequeuniversity.com/rules/axe/4.10/autocomplete-valid?application=playwright', nodes: [ { any: [], all: [Array], none: [], impact: 'serious', html: '', target: [Array], failureSummary: 'Fix all of the following:\n' + ' the autocomplete attribute is incorrectly formatted' }, { any: [], all: [Array], none: [], impact: 'serious', html: '', target: [Array], failureSummary: 'Fix all of the following:\n' + ' the autocomplete attribute is incorrectly formatted' } ] } ```

issue with share side panel

``` { id: 'color-contrast', impact: 'serious', tags: [ 'cat.color', 'wcag2aa', 'wcag143', 'TTv5', 'TT13.c', 'EN-301-549', 'EN-9.1.4.3', 'ACT' ], description: 'Ensure the contrast between foreground and background colors meets WCAG 2 AA minimum contrast ratio thresholds', help: 'Elements must meet minimum color contrast ratio thresholds', helpUrl: 'https://dequeuniversity.com/rules/axe/4.10/color-contrast?application=playwright', nodes: [ { any: [Array], all: [], none: [], impact: 'serious', html: 'Add link', target: [Array], failureSummary: 'Fix any of the following:\n' + ' Element has insufficient color contrast of 4.19 (foreground color: #4a76ac, background color: #edf3fa, font size: 10.5pt (14px), font weight: normal). Expected contrast ratio of 4.5:1' } ] } ```

issue with breadcrumbs

``` { id: 'aria-hidden-focus', impact: 'serious', tags: [ 'cat.name-role-value', 'wcag2a', 'wcag412', 'TTv5', 'TT6.a', 'EN-301-549', 'EN-9.4.1.2' ], description: 'Ensure aria-hidden elements are not focusable nor contain focusable elements', help: 'ARIA hidden element must not be focusable or contain focusable elements', helpUrl: 'https://dequeuniversity.com/rules/axe/4.10/aria-hidden-focus?application=playwright', nodes: [ { any: [], all: [Array], none: [], impact: 'serious', html: '

', target: [Array], failureSummary: 'Fix all of the following:\n' + ' Focusable content should have tabindex="-1" or be removed from the DOM' } ] } ```