Investigate intermittent GET /matrix 504s during functional tests

[calvinnhieu]

API Gateway metrics capture 5XX errors, the majority of which have been traced to be 504s with the exception of frequent 500s thrown in integration due to two hanging DSS ES subscriptions (now deleted and resolved). The remaining large spikes in 5XX errors can often be attributed to scheduled Redshift maintenance/upgrades in which the cluster is down for several minutes.

What remains unexplained are the remaining spikes that do not correlate to either of these events in which 504s are thrown on GET /matrix during functional tests causing intermittently failing GitLab builds in all environments. Determine the root cause of these errors.