Checks created via the API (terraform provider via Pulumi) don't show up until a check is manually added or deleted in the UI

[cnemo-cenic]

What happened:

In a Grafana Cloud environment.

I created several checks via the API, using Pulumi. These checks ran successfully for several hours, and showed up on the synthetics "Home" page [1]. However, on the "Checks" page [2], I could not see the checks at all. And on the check-specific [3] pages, I saw the error "Check not found" "The check you're trying to view does not exist. Try the checks page instead.". Once I went back to the "Checks" page and manually added or deleted an unrelated check, all of the API-created checks showed up and the check-specific pages no longer returned the not found error.

[1] /a/grafana-synthetic-monitoring-app/home
[2] /a/grafana-synthetic-monitoring-app/checks
[3] /a/grafana-synthetic-monitoring-app/checks/

What you expected to happen: Checks created via Pulumi show up in the app correctly without intervention.

How to reproduce it (as minimally and precisely as possible): See above

Anything else we need to know?:

Environment: Grafana Cloud

• Grafana version (if running on prem):
• SM plugin version:
• OS Grafana is installed on:
• User OS & Browser:
• Others:

[peterschretlen]

Thank you for logging this issue @cnemo-cenic. The reason this happens is that the synthetic monitoring plugin does not refresh it's cache of checks (which involves calling an API) when doing many actions within the app. Meanwhile the homepage does not use API calls at all: it relies on datasources (logs, metrics, and a synthetic monitoring datasource). This means the data shown in the homepage can be out of sync with the data cached in app.

A refresh of the page will usually solve this by forcing the app to reload the list of checks. But we will consider invalidating the cache on an interval to at least alleviate this problem. To properly fix it we'd need to push the changes in real-time to Grafana, perhaps with a websocket connection or something along those lines.