Skip to content

Cross browser Accessibility Platform Testsuite Web Engines Hackfest 2021

Jump to bottom Edit New page
Manuel Rego Casasnovas edited this page May 12, 2021 · 2 revisions

Cross-browser Accessibility Platform Testsuite - Web Engines Hackfest 2021

Minutes

[Brian K (Igalia)] Let's get started! See IRC and Matrix rooms to participate in sidechannel conversation. We hope they can substitute for in-person conversation.

[Brian] We'll have a presentation and then a discussion I'll moderate. We'll do our best to keep notes and minutes of that discussion, but it won't be recorded. Any questions before we get started?

(Presentation by Alex Surkov)

[Brian] We’ll try to let people weight in and speak. Please try to mute yourself when not speaking.

[Dominic M (Google)] I think it would be pretty cool if the work we did to make Chromium more testable helps. It’s great for those of us working on Chromium that the code depends on Chromium, but it feels daunting for people working in other browsers. We’ve thought about splitting out core accessibility code into its own module. I’d imagine splitting out the cross-platform interfaces to create a native accessibility object. It’s only a tiny piece of whatæs needed, but it abstracts the platform as much as possible. Properties and attributes are well abstracted by Chromium, but events aren’t. If this gets separated into another project on GitHub, it could slow down the development speed of Chromium itself. So we’d want more buy-in from others before taking that step.

[Brian] You said you’ve thought about some of this before. Has it been discussed with WebKit or Gecko contributors?

[Dominic] Not this specific issue of a cross-platform accessibility test suite. We did talk about ATTA. The way it came up before is another Google project was building a desktop application and needed accessibility and wanted to borrow Chrome’s code. Which we couldn’t do easily. If there was enough interest from other projects, we could pull Chromium accessibility into its own module for others to use.

[Brian] It’s a really intriguing idea. The more we collaborate and test together, the more we tend to all benefit in lots of ways.

[Andres G (Apple)] A good first step would be to have a suite of test cases. It’s an attainable initial goal to have tests for all the different engines.

[Brian] So a WPT equivalent.

[Andres] Right.

[Alex S] We could start from ARIA tests.

[Andres] Agreed.

[Brian] It makes sense to start with small agreements and get the ball rolling.

[Dominic] Joanie, I’d love to hear your experience with ATTA.

[Joanie (Igalia)] My first comment is to chuckle about the ATTA project being big; it wasn’t. We did all our CR conformance testing manually. We wrote a fake AT with event listeners and hooked it up with a WPT hack and then everyone did their own thing. It was just meant to hit exit criteria, it was never meant to be a cross-platform solution. More hands are needed. One thing I’m hoping to do is stop ATTA and create something better to take its place, based on Chromium's tools (with Google's OK).

[Dominic] It seemed big to me because nobody had ever tried to build something that size before.

[Joanie] Because of the hacky nature, the current tests require you have a whole bespoke setup and it asks for a role and goes and gets the test. They’re manual because not everyone has the needed ATTAs installed. If we had a quality tool that could either hook into WPT or some other automated tool, what I’d then like to do is port the accname tests into the accname spec repo, they were ran and updated manually and we recently ported them into the Chromium test suite, where they are run periodically. That could serve as part of the test, can cover ARIA and HTML AAM tests. If we can split this out, it could be run in any browser. On my back burner, I’d like to create Orca-based dump browser tests.

[Brian] It sounds like from a W3C standpoint, there’s a recognized need for something. Which is promising in the sense that there’s a proven need. So what do we need to do to get agreement to advance toward that something?

[Alex S] We could share this talk and the ideas in it. Hopefully that could be a starting point for further conversation.

[Brian] Do we have agreement that WebKit’s feedback about starting with a small universal test suite would be useful?

[Dominic] It’s more a question of, how do we get there? Right now we’ve built up what we have in Chromium out of a desire to fix our own bugs, and to make the tests as honest as possible using platform APIs. There’s still a large gap between what we can do now and what would make a good cross-platform test suite. Who has the time and motivation?

[Brian] Yeah, it sounded like the first step would be to show proof of interest with a small test suite. It doesn’t tell us we ultimately arrive at the end goal, but it lets us take a step.

[Dominic] My experience is to make something stick, make it part of continuous integration. What would it take to have something like this be part of Apple or Mozilla’s tests? Would it be a hard requirement to build from source? It would be a hard battle internally to say to my team, let’s take a binary someone else wrote and make it part of our tests. But if it were source code, that’s an easier sell.

[Andres] That sounds reasonable.

[Jacobo (Igalia)] Another way to do this would be to have a third party create the test suite. Something periodically checked Chromium, Firefox, WebKit, etc., and then sending out results.

[Brian] Are y ou suggesting a new third-party service would check out a shared test suite, like a bot, and run it on each of the engines?

[Jacobo] Dominic was asked how to make the tests part of the browsers’ process. Maybe if we had a thing that checked in an agreed and growing test suite and run the suite in browser builds and then sent out or published the results.

[Joanie] I’m all in favor of the idea. All the ARIA specs have integration features. Hypothetically, what if the W3C had a bot that ran all the tests any time there was a spec change? What I want to do is get the ARIA tests into the spec repository. Could we do this on W3C infrastructure?

[Brian] Maybe? There’s a lot of cross-use of different things.

[Joanie] It would also solve the problem of the Working Group always wondering what’s supported and what’s changed.

[Brian] That is a really interesting idea.

[Dominic] My opinion is that tests are infinitely more useful when they’re run at build time. If they’re run elsewhere, they’re not useful when someone’s trying to refactor or build things. I really feel like what’s needed is something that browsers can run the same tests.

[Brian] We could do both, right?

[Dominic] Sure, yes. Maybe when new tests are added they’d be run in a central place before being distributed out to browsers.

[Joanie] I don’t disagree. But this is one of the reasons I think of having a single repository that we all pull from. If an implementation changes but that doesn’t get back into the tests, then the tests can start failing. Also, different browsers have different ideas about with the accessibility tree should be. We need a way to minimize those differences.

[Brian] That happens with WPT as well, where edge cases run into disagreement. This is what I meant by having a common language to discuss these things. Sometimes, what the WG write leaves too much open to interpretation, and we find that out when we create tests. Once you have a test, you can discuss why it was done that way. With the tests, we’re more likely to get agreement on what the specification means. There are benefits to a periodic report; it gets us things like CanIUse and WPT.fyi. Getting the tests into the build processes is also important. But we have to start somewhere! So a small set of tests would be a great starting point.

[Joanie] Can we do that with the accname tests that are in Chromium’s test suite now? They were imported from the working group test expectations. That seems to me like a good place to start.

[Alex] I support that idea.

[Brian] How does that sound to other browser vendors?

[Andres] Where are those tests?

[Joanie] They’re part of the tests Alex was discussing.

[Andres] We also have some tests of computer accessibility names internally. Would like to see what gap there might be.

[Dominic] I think we’re talking about using command line tools that can be run outside of Chromium’s tests. All the test code in WebKit as far I know requires running within that environment. We’re proposing a standalone CLI tool that is based on Chromium, but you can run it against any browser.

[Joanie] I added links in the chat to the tests and expectations. There you have, for example, for the first test named "desc-combobox-focusable", expectation files for several platforms (txt files) and the html that generates them.

[Alex] I was thinking to generalize things so we could have support on multiple platforms.

[Joanie] I trust you that it makes sense. I don’t fully understand it myself.

[Andres] We’re talking about defining the tests, and then how to run them on different browsers on different platforms. So I see it as two steps. Coming up with the HTML and CSS code that generates the test cases that’s not platform dependent, and then we can think about how to run those tests on each browser and how they integrate into build process.

[Brian] If we started with the accname tests, then pretty soon we’d be able to run those tests in browser build processes and we could set up a reporting process.

[Dominic] Agreed. Yeah agreeing on a set of HTML+CSS+JS to run is kind of the easy part. It’s the expectations generated when you run the test that are hard to share. Because you have to agree on the abstraction. accname might be a great place to start because the output is one string.

[Brian] I think it sounds really interesting and we can get something that we have some immediate benefits. Then we can launch into the more nuanced a difficult conversations from that. It’s true this is the easy part, but that’s often a good place to start, because it gives you momentum.

[Dominic] Agreed, if we have something we could run against browsers, then maybe the next step is static pages we can test against, and then later dynamic tests, which are the hardest. And also where most of the bugs are.

[Brian] Maybe we don’t wait to think about those harder things, but we can talk about them in parallel with the easy stuff. This has been a great discussion, but I do want to mindful of the time. Anyone have something they want to bring up?

[Alex] A demo of something that works on browsers other than Chromium would be good as a proof of concept.

[Brian] We can stay on for a few more if people like, or if everyone’s done, we can wrap up.

(thanks and goodbye)

https://www.webengineshackfest.org/

Add a custom sidebar
Clone this wiki locally