[GCP] Intermittent failure for gfx-unit-test-tool

[jkwak-work]

GCP server shows higher intermittent failure rate than expected.

We need to investigate how to improve it.

[jkwak-work]

The failing tests are almost always gfx-unit-test-tool/.
Unlike other tests, gfx-unit-tests don't get a chance to be retried at the end.
We should improve it to make GCP more stable.

[jkwak-work]

I created a PR to test CI with "-server-count 1", #6908

I will let it run for 10 rounds and see if it help.

[jkwak-work]

When the intermittent failure is seen, I observed the following error message,

[Failed]: !SLANG_FAILED(device->createBufferResource(bufferDesc, initialData, outBuffer.writeRef()))

[jkwak-work]

I am not sure how to read it but github has a statistics for the CI runs.
https://github.com/shader-slang/slang-rhi/actions/metrics/performance?dateRangeType=DATE_RANGE_TYPE_CURRENT_WEEK&sort=failureRate&tab=jobs

[jkwak-work]

We need to implement to retry when gfx-unit-test fails.
The previous implementation can be found here

• slang-test: retry failed test at the end. #5255

[jkwak-work]

Based on a few tests, it looks like the high rate of failure of CI is due to the parallel execution of slang-test.
When the server count is set to 1, no failure is observed for five consecutive runs.

• [DNI] Test/test ci #6908

@aasgaonkar also reported that when the server count is set to 4, no failure was observed for five runs.

• reduce server count to 4 for testing #6910

An ideal solution for the issue is to implement the retry logic to "XXX-unit-test-tool/" tests.
When the retry is applied to those tests, 10 consecutive runs didn't fail on Windows/Release, although the PR seems to have a problem on Linux.

• Retry when a few unit tests failed. #6912

Closing the issue and we will track it on the other issue.

• Apply retry logic to slang-unit-test-tool and gfx-unit-test-tool #6926