Update checks, creating new ones instead of updating the existing job check

[r0x0d]

Hello!

In Convert2RHEL, we noticed that the checks for this action are not updating the job itself, but rather, they are creating new ones when the job fails.

We wanted to know if that is the correct behavior, or we are setting something wrong in our usage. We would expect that this action would update the job itself with the status, instead of creating a new one and having a confuse status report under the PR… Currently, we have the job that was executed marked as green, and another check with a red mark related to the same job that ran.

Link for the PR: oamg/convert2rhel#528

The image might not show much, but almost all tests failed in this PR, and if you look in the checks, you will notice that we have it duplicated, one with a green mark because the job finished and one with the red mark because the job failed.

[bocekm]

The current behavior @r0x0d explained has another negative side effect. When opening the PR's Checks tab we see all tests as green/passing even if they in fact failed. This is not only misleading but also we can't re-run just the failed tests. Normally there's a button for that in GitHub if some of the tests are reported as failed but in this case the button is not there as the failed tests don't appear there at all.

[zmiklank]

Hi, thank you for creating the issue.

The current behavior @r0x0d explained has another negative side effect. When opening the PR's Checks tab we see all tests as green/passing even if they in fact failed. This is not only misleading but also we can't re-run just the failed tests. There's a button for that in GitHub if some of the tests are reported as failed.

This is intended. When a test fails, its status is red (failed), but overall result of the job is green (meaning all tests have been run and action finished as was expected). In case of some error in the action the overal result of the action is fail - red.
Question is what is the correct approach here. @phracek, any opinion?

In Convert2RHEL, we noticed that the checks for this action are not updating the job itself, but rather, they are creating new ones when the job fails.

I did not run into this issue when running our actions base on testing-farm-as-a-github-action. I need to take closer look at what has happened in your case.

[zmiklank]

Hi, I think that the double status is caused by the chained calling of the actions in the convert2rhel repository.
IMO firstly the tmt-tests.yml is called which creates first status (https://github.com/oamg/convert2rhel/blob/9a7f1032dcd7df4b18ae7b8415dfa713a0eecae7/.github/workflows/tmt-tests.yml#L86), which is in this case green, as it end with success - reuse-int-tests.yml is called successfully.
Then the testing-farm-as-github-action create its own status - in this case red, as tests failed.

Could that be it?

[r0x0d]

Hi, @zmiklank! That seems like a very plausible cause… I was wondering if there is no way to tell testing-farm-as-github-action to update an existing check instead of creating a new one.

Not sure how that would work with reusable jobs like this, but, that would be super nice to have.