False positive when percy is failing

[Maxwell2022]

Not sure if this is a Percy CLI issue or a problem with this action or maybe just a configuration missing somehow.

While migrating our pipeline to Github Action, we switch to this cypress action to run our E2E suite in parallel (using Cypress Cloud).

We noticed that sometime the Percy check was not showing in the PR status check. We looked at the action logs only to notice that when Percy was failing in some random jobs and not actually exiting the test execution with an error code.

For instance we had a parallel run with one of the job that output this but did not fail:

[percy] Skipping visual tests
[percy] Error: Nonce is too long (maximum is 64 characters)

another with

[percy] Detected error for percy build
[percy] Failure: Browser launch failure
[percy] Failure Reason: Percy was not able to launch a browser on your machine to capture snapshots.
[percy] Suggestion: Please resolve the error faced while launching the browser: . Try setting PERCY_BROWSER_EXECUTABLE to the correct Chrome browser executable path.
[percy] Refer to the below Doc Links for the same
[percy] * https://www.browserstack.com/docs/percy/common-issue/common-errors/browser-launch-failure
[percy] Skipping visual tests
[percy] Error: Failed to launch browser. Timed out after 30000ms
[5892:5892:1030/062153.679291:ERROR:base/allocator/scheduler_loop_quarantine_config.cc:129] No entry found.
[5892:5892:1030/062153.679358:ERROR:base/allocator/scheduler_loop_quarantine_config.cc:129] No entry found.
[5892:5892:1030/062153.700217:ERROR:base/allocator/scheduler_loop_quarantine_config.cc:129] No entry found.
[5892:5925:1030/062156.920850:ERROR:dbus/bus.cc:408] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are “tcp” and on UNIX “unix”)

Apart from the face these errors are most likely genuine, we would have expected the test to fail at this point instead to continue and run the cypress suite.

Our action looks like that:

jobs:
  e2e-tests:
    name: 🧑‍🔬 E2E Tests
    runs-on: 'xxx'
    timeout-minutes: 40

    strategy:
      fail-fast: false
      matrix:
        containers: [1, 2, 3, 4]


    steps:
      - name: 📦 Checkout code
        uses: actions/checkout@v5


      [...]


      - name: 🤖 Install Cypress Binaries
        uses: ./.github/actions/configure-cypress

      - name: 👩‍🔬 Cypress run
        uses: cypress-io/github-action@v6
        env:
          COMMIT_INFO_MESSAGE: ${{ github.event.pull_request.title || github.event.head_commit.message }}
          COMMIT_INFO_BRANCH: ${{ github.head_ref }}
          CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          PERCY_BRANCH: ${{ github.head_ref }}
          PERCY_BROWSER_EXECUTABLE: /usr/bin/chromium-browser
          PERCY_CLIENT_ERROR_LOGS: 'false'
          PERCY_PARALLEL_NONCE: ${{ github.event_name }}-${{ github.run_id }}-${{ github.run_attempt }}
          PERCY_PARALLEL_TOTAL: ${{ strategy.job-total }}
          PERCY_TARGET_BRANCH: master
          PERCY_TOKEN: ${{ secrets.PERCY_TOKEN }}
        with:
          install: false
          record: true
          parallel: true
          group: 'E2E'
          command-prefix: 'percy exec -t 60 -- pnpm'

      - name: ...

Any help of direction would be appreciated. Thanks

[MikeMcC399]

@Maxwell2022

You describe an issue where you are using the action with parameters:

          install: false
          record: true
          parallel: true
          command-prefix: 'percy exec -t 60 -- pnpm'

When you see error messages from Percy, you expect that the GitHub Actions job would show a failure, however that is not happening.

Unfortunately integration with Percy is not set up for testing within the actions' own CI, so we don't have a reference installation to compare with or to do testing on.

You may be able to find out more in your own environment by enabling debugging.

To rule out whether load distribution through Cypress Cloud parallel recording is related to the issue, you could disable the matrix, disable record and disable parallel options.

[Maxwell2022]

Update:

We are using percy/percy-cypress and configured cypress with percyThrowErrorOnFailure: true

However, this is still happening, we can see errors from Percy but the job is still successful.

[percy] Detected error for percy build
[percy] Failure: Browser launch failure
[percy] Failure Reason: Percy was not able to launch a browser on your machine to capture snapshots.
[percy] Suggestion: Please resolve the error faced while launching the browser: . Try setting PERCY_BROWSER_EXECUTABLE to the correct Chrome browser executable path.
[percy] Refer to the below Doc Links for the same
[percy] * https://www.browserstack.com/docs/percy/common-issue/common-errors/browser-launch-failure
[percy] Skipping visual tests
[percy] Error: Failed to launch browser. Timed out after 30000ms
[5433:5433:1102/225838.015763:ERROR:base/allocator/scheduler_loop_quarantine_config.cc:129] No entry found.
[5433:5433:1102/225838.015813:ERROR:base/allocator/scheduler_loop_quarantine_config.cc:129] No entry found.
[5433:5433:1102/225838.037813:ERROR:base/allocator/scheduler_loop_quarantine_config.cc:129] No entry found.
[5433:5497:1102/225841.301889:ERROR:dbus/bus.cc:408] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
[5518:5518:1102/225842.345481:ERROR:base/allocator/scheduler_loop_quarantine_config.cc:129] No entry found.
[5518:5518:1102/225842.345616:ERROR:base/allocator/scheduler_loop_quarantine_config.cc:129] No entry found.
[5538:5538:1102/225845.908047:ERROR:base/allocator/scheduler_loop_quarantine_config.cc:129] No entry found.
[5538:5538:1102/225845.908089:ERROR:base/allocator/scheduler_loop_quarantine_config.cc:129] No entry found.
[5556:5556:1102/225849.759088:ERROR:base/allocator/scheduler_loop_quarantine_config.cc:129] No entry found.
[5556:5556:1102/225849.759213:ERROR:base/allocator/scheduler_loop_quarantine_config.cc:129] No entry found.
[5433:5497:1102/225849.919351:ERROR:dbus/bus.cc:408] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")

[MikeMcC399]

@Maxwell2022

The information in this issue does not yet allow any explicit diagnosis. If you are able to provide a public repo which demonstrates the issue, we can take a look.

You may also want to try running Percy / Cypress without using the action. Since you are using the action with the option install: false, you are already bypassing installation and caching functions.

Another possibility would be to report your issue to @percy/cypress.