Pa11y CI returns no errors when known errors are present

[maggiewachs]

I use several tools to evaluate my project's websites, and often use Pa11y CI to validate them. Recently I noticed that when other tools are picking up errors, Pa11y CI reports none. It looks like Pa11y CI is running older versions of Pa11y, axe, and Puppeteer, could that be related? The latest update re: dependabot never merged: #244

For example, this morning I tested a single page locally (in the command line) using Pa11y CI, Pa11y, and axe-core/cli, all with default configurations, and got these results (see detailed log below):

• Pa11y CI (html_cs): 0 errors
• Pa11y (html_cs): 12 errors
• axe: 3 errors

Last login: Thu Jan  2 08:13:41 on ttys000
MacBook-Pro ~ % pa11y-ci https://www.healthcare.gov/
(node:23267) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
Running Pa11y on 1 URLs:
 > https://www.healthcare.gov/ - 0 errors

✔ 1/1 URLs passed
MacBook-Pro ~ % pa11y https://www.healthcare.gov/

Welcome to Pa11y

 > Running Pa11y on URL https://www.healthcare.gov/

Results for URL: https://www.healthcare.gov/

 • Error: Img element with empty alt text must have absent or empty title attribute.
   ├── WCAG2AA.Principle1.Guideline1_1.1_1_1.H67.1
   ├── #gatsby-focus-wrapper > div > div:nth-child(1) > header > div:nth-child(2) > nav > div > div > div > div:nth-child(4) > div > img
   └── <img src="/assets/learn/static/featured/oe-ends-dec-16-featured-1.jpg" alt="" class="megaMenu-module--featured-image--63c97" title="Woman pointing to clock">

 • Error: Img element with empty alt text must have absent or empty title attribute.
   ├── WCAG2AA.Principle1.Guideline1_1.1_1_1.H67.1
   ├── #main > div:nth-child(2) > section:nth-child(1) > div > div:nth-child(1) > div > img
   └── <img src="/assets/learn/static/hero/oe12-homepage-oe-ends-jan15-deploy-dec16-hero.png" alt="" class="index-module--hero-image--84dbc" title="Two people standing back to back arms crossed">

 • Error: Img element with empty alt text must have absent or empty title attribute.
   ├── WCAG2AA.Principle1.Guideline1_1.1_1_1.H67.1
   ├── #main > div:nth-child(2) > section:nth-child(3) > div:nth-child(1) > img
   └── <img src="/assets/learn/static/featured/oe-ends-dec-16-featured-1.jpg" alt="" class="FeaturedItem-module--media--88086" title="Woman pointing to clock">

 • Error: Img element with empty alt text must have absent or empty title attribute.
   ├── WCAG2AA.Principle1.Guideline1_1.1_1_1.H67.1
   ├── #main > div:nth-child(2) > section:nth-child(3) > div:nth-child(2) > img
   └── <img src="/assets/learn/static/featured/last-chance-jan-1-feature2.jpg" alt="" class="FeaturedItem-module--media--88086" title="Couple looking at their computer smiling">

 • Error: Img element with empty alt text must have absent or empty title attribute.
   ├── WCAG2AA.Principle1.Guideline1_1.1_1_1.H67.1
   ├── #main > div:nth-child(2) > section:nth-child(3) > div:nth-child(3) > img
   └── <img src="/assets/learn/static/featured/oe-starts-topics-image3.jpg" alt="" class="FeaturedItem-module--media--88086" title="Person with glasses paying online with credit card">

 • Error: This form does not contain a submit button, which creates issues for those who cannot submit the form using the keyboard. Submit buttons are INPUT elements with type attribute "submit" or "image", or BUTTON elements with type "submit" or omitted/invalid.
   ├── WCAG2AA.Principle3.Guideline3_2.3_2_2.H32.2
   ├── #main > div:nth-child(2) > div > div > section:nth-child(1) > form
   └── <form><div class="ds-u-clearfix Impor...</form>

 • Error: Iframe element requires a non-empty title attribute that identifies the frame.
   ├── WCAG2AA.Principle2.Guideline2_4.2_4_1.H64.1
   ├── #utag_323_iframe
   └── <iframe id="utag_323_iframe" height="1" width="1" style="display:none" src="//8209133.fls.doubleclick.net/activityi;src=8209133;type=2022;cat=2022-006;ord=7092714912537.379?"></iframe>

 • Error: Iframe element requires a non-empty title attribute that identifies the frame.
   ├── WCAG2AA.Principle2.Guideline2_4.2_4_1.H64.1
   ├── #utag_338_iframe
   └── <iframe id="utag_338_iframe" height="1" width="1" style="display:none" src="//8966771.fls.doubleclick.net/activityi;src=8966771;type=invmedia;cat=cms-20;ord=5281145606721.596?"></iframe>

 • Error: Iframe element requires a non-empty title attribute that identifies the frame.
   ├── WCAG2AA.Principle2.Guideline2_4.2_4_1.H64.1
   ├── #utag_386_iframe
   └── <iframe id="utag_386_iframe" height="1" width="1" style="display:none" src="//8966771.fls.doubleclick.net/activityi;src=8966771;type=invmedia;cat=cms-u0;qty=1;cost=;ord=1735920321270?"></iframe>

 • Error: Img element missing an alt attribute. Use the alt attribute to specify a short text alternative.
   ├── WCAG2AA.Principle1.Guideline1_1.1_1_1.H37
   ├── #top > div:nth-child(20) > div:nth-child(1) > div > img
   └── <img src="https://gov1.qualtrics.com/WRQualtricsSiteIntercept/Graphic.php?IM=IM_brriXYMu68CEU3s" style="width: 790px; height: 540px;">

 • Error: Img element missing an alt attribute. Use the alt attribute to specify a short text alternative.
   ├── WCAG2AA.Principle1.Guideline1_1.1_1_1.H37
   ├── #top > div:nth-child(20) > div:nth-child(2) > div > img
   └── <img src="https://gov1.siteintercept.qualtrics.com/static/q-siteintercept/~/img/svg-close-btn-white-1.svg">

 • Error: Iframe element requires a non-empty title attribute that identifies the frame.
   ├── WCAG2AA.Principle2.Guideline2_4.2_4_1.H64.1
   ├── html > b > iframe
   └── <iframe style="display: none;"></iframe>

12 Errors

MacBook-Pro ~ % axe https://www.healthcare.gov/
Running axe-core 4.10.2 in chrome-headless

Testing https://www.healthcare.gov/ ... please wait, this may take a minute.

  Violation of "link-name" with 3 occurrences!
    Ensure links have discernible text. Correct invalid elements at:
     - .ds-c-external-link.connectWithUs-module--icon--80195.external-link-hide-icon-override:nth-child(3)
     - .ds-c-external-link.connectWithUs-module--icon--80195.external-link-hide-icon-override:nth-child(5)
     - .ds-c-external-link.connectWithUs-module--icon--80195.external-link-hide-icon-override:nth-child(7)
    For details, see: https://dequeuniversity.com/rules/axe/4.10/link-name

3 Accessibility issues detected.
Testing complete of 1 pages

[aarongoldenthal]

I think you've hit the issue - outdated dependencies. At the moment pa11y is running with some old versions of puppeteer and axe-core, and pa11y-ci is running some very very old versions puppeteer and pa11y (the last published release of pa11y-ci, for example, is running puppeteer v9, which uses Chromium 91 from 2021, and the current is puppeteer v23 with Chromium 131).

With the stagnation in updates, I've been publishing my forks of pa11y and pa11y-ci - at this point primarily focused on significant dependency (for example, puppeteer and axe-core, which impact the analysis) and security updates. If you run them for comparison, you do see the same results with between pa11y and pa11y-ci (for your example with https://www.healthcare.gov/, 6 errors with htmlcs, 22 errors with axe).

[maggiewachs]

@aarongoldenthal thanks for the update/reply. At this point I may just move over to using axe-core with puppeteer since my implementation of Pa11y CI is pretty simple. But I'd like to see this project continue since it's been so valuable in getting folks to implement testing in their pipelines.

If the maintainers see this issue, can we get an update on long term plans?