We should:

1. Ensure a Slack channel is notified when CLI benchmark tests fail
2. See if the metabase dashboards can turn "red" (or have some other visual indication) when tests are failing

From @t0yv0:

there is a marker I think in the data though that talks to the run having failed; perhaps some sql munging can change the presentation

The details are - there's basically some accidental coupling in the data harvester that is not robust to individual tests failing so it prefers them. not to fail if some are failing which is most of the time. It's not unreasonable if the success/failure status is written to Redshift like i suspect.

Related:

• CLI performance metrics examples broken due to AWS S3 security changes #1430
• CLI performance metrics TestManyResources examples broken #1431