[ISSUE] Random 503 'Service Temporarily Unavailable' Errors in EKS QA Test Clusters

[slolatte]

Describe the issue:

We have a few workflows which have been migrated to use the Infra EKS cluster. However, we've noticed some stability issues with these environments. For example, we often see 503 Service Temporarily Unavailable errors. The latest environment where we have observed this instability was generated here.

Slack discussion can be seen here.

Actual behavior:

During a test run, our tests are failing due to Console becoming unhealthy randomly and seeing 503 'Service Temporarily Unavailable' Errors.

Expected behavior:

None of this instability should be occurring.

How to reproduce:

Generate any QA HC using the EKS cluster and go onto the generated test environment.

Logs:

Environment:

Please note: Without the following info, it's hard to resolve the issue and probably it will be closed.

• Platform:
• Helm CLI version:
• Chart version:
• Values file:

[eamonnmoloney]

We have made some changes to that cluster since this was created. @slolatte can this be closed for now until we get back to deploying into EKS and log specific issues we see now?

[slolatte]

@eamonnmoloney Yes this makes sense to me, I'll close this ticket.