Skipped checks on automerge are falsly considered successful

[evantahler]

We've got classic branch protections enabled on our main branch.

Our test suite is set up such that a number of sub-jobs need all pass which then trigger a final job, backend-test-complete, which is the required check in the branch protection.

jobs: 

...

  backend-test-complete:
    runs-on: ubuntu-latest
    needs:
      - backend-lint
      - backend-seed
      - backend-tests
      - backend-acceptance-tests
    steps:
      - run: echo "SUCCESS!"

If one of the sub jobs fails (in this example, backend-seed), the final job, backend-test-complete is skipped.

However, auto-merge proceeded when it should have been prevented. The language about branch protections implies that "skipped" jobs should not pass the branch protection, and only "successful" jobs should pass:

So... I think it is a bug that automerge happened in this case. Thanks!