`dugtrio_pool_online` metric doesn't correlate with log info

[dpulitano]

We have an alert set up that will warn us if 1/2 Beacon RPCs is unavailable for more than 5 minutes

We are reading the dugtrio_pool_online metric in order to alert on this. Seeing the PromQL query, max_over_time(dugtrio_pool_online{container_name="$Network"}[5m]), go down to 1 11 times in 24 hours:

However, based on the application logs for the last 24 hours, each time a provider becomes unavailable, it recovers within 1 minute. So the PromQL query should have never dropped down to 1:

2025-05-09 15:31:45.863	time="2025-05-09T15:31:45Z" level=info msg="status changed  Online -> Offline" client=eth-sepolia-backend-1
2025-05-09 14:31:49.703	time="2025-05-09T14:31:49Z" level=info msg="status changed  Offline -> Online" client=eth-sepolia-backend-1
2025-05-09 14:31:42.859	time="2025-05-09T14:31:42Z" level=info msg="status changed  Online -> Offline" client=eth-sepolia-backend-1
2025-05-09 13:31:48.352	time="2025-05-09T13:31:48Z" level=info msg="status changed  Offline -> Online" client=eth-sepolia-backend-1
2025-05-09 13:31:39.857	time="2025-05-09T13:31:39Z" level=info msg="status changed  Online -> Offline" client=eth-sepolia-backend-1
2025-05-09 12:31:48.470	time="2025-05-09T12:31:48Z" level=info msg="status changed  Offline -> Online" client=eth-sepolia-backend-1
2025-05-09 12:31:36.853	time="2025-05-09T12:31:36Z" level=info msg="status changed  Online -> Offline" client=eth-sepolia-backend-1
2025-05-09 11:31:48.392	time="2025-05-09T11:31:48Z" level=info msg="status changed  Offline -> Online" client=eth-sepolia-backend-1
2025-05-09 11:31:33.849	time="2025-05-09T11:31:33Z" level=info msg="status changed  Online -> Offline" client=eth-sepolia-backend-1
2025-05-09 10:31:36.822	time="2025-05-09T10:31:36Z" level=info msg="status changed  Offline -> Online" client=eth-sepolia-backend-1
2025-05-09 10:31:30.847	time="2025-05-09T10:31:30Z" level=info msg="status changed  Online -> Offline" client=eth-sepolia-backend-1
2025-05-09 09:31:36.345	time="2025-05-09T09:31:36Z" level=info msg="status changed  Offline -> Online" client=eth-sepolia-backend-1
2025-05-09 09:31:27.845	time="2025-05-09T09:31:27Z" level=info msg="status changed  Online -> Offline" client=eth-sepolia-backend-1
2025-05-09 09:27:51.637	time="2025-05-09T09:27:51Z" level=info msg="status changed  Offline -> Online" client=eth-sepolia-backend-2
2025-05-09 09:27:48.612	time="2025-05-09T09:27:48Z" level=info msg="status changed  Online -> Offline" client=eth-sepolia-backend-2
2025-05-09 08:31:36.688	time="2025-05-09T08:31:36Z" level=info msg="status changed  Offline -> Online" client=eth-sepolia-backend-1
2025-05-09 08:31:24.842	time="2025-05-09T08:31:24Z" level=info msg="status changed  Online -> Offline" client=eth-sepolia-backend-1
2025-05-09 07:31:36.299	time="2025-05-09T07:31:36Z" level=info msg="status changed  Offline -> Online" client=eth-sepolia-backend-1
2025-05-09 07:31:21.840	time="2025-05-09T07:31:21Z" level=info msg="status changed  Online -> Offline" client=eth-sepolia-backend-1
2025-05-09 06:31:24.718	time="2025-05-09T06:31:24Z" level=info msg="status changed  Offline -> Online" client=eth-sepolia-backend-1
2025-05-09 06:31:18.837	time="2025-05-09T06:31:18Z" level=info msg="status changed  Online -> Offline" client=eth-sepolia-backend-1
2025-05-09 05:31:24.378	time="2025-05-09T05:31:24Z" level=info msg="status changed  Offline -> Online" client=eth-sepolia-backend-1
2025-05-09 05:31:15.834	time="2025-05-09T05:31:15Z" level=info msg="status changed  Online -> Offline" client=eth-sepolia-backend-1
2025-05-09 04:31:25.693	time="2025-05-09T04:31:25Z" level=info msg="status changed  Offline -> Online" client=eth-sepolia-backend-1
2025-05-09 04:31:12.831	time="2025-05-09T04:31:12Z" level=info msg="status changed  Online -> Offline" client=eth-sepolia-backend-1
2025-05-09 03:31:13.026	time="2025-05-09T03:31:13Z" level=info msg="status changed  Offline -> Online" client=eth-sepolia-backend-1
2025-05-09 03:31:09.828	time="2025-05-09T03:31:09Z" level=info msg="status changed  Online -> Offline" client=eth-sepolia-backend-1
2025-05-09 02:31:12.729	time="2025-05-09T02:31:12Z" level=info msg="status changed  Offline -> Online" client=eth-sepolia-backend-1
2025-05-09 02:31:06.825	time="2025-05-09T02:31:06Z" level=info msg="status changed  Online -> Offline" client=eth-sepolia-backend-1
2025-05-09 01:31:11.502	time="2025-05-09T01:31:11Z" level=info msg="status changed  Offline -> Online" client=eth-sepolia-backend-1
2025-05-09 01:31:03.822	time="2025-05-09T01:31:03Z" level=info msg="status changed  Online -> Offline" client=eth-sepolia-backend-1
2025-05-09 00:31:14.093	time="2025-05-09T00:31:14Z" level=info msg="status changed  Offline -> Online" client=eth-sepolia-backend-1
2025-05-09 00:31:00.818	time="2025-05-09T00:31:00Z" level=info msg="status changed  Online -> Offline" client=eth-sepolia-backend-1
2025-05-08 23:31:12.653	time="2025-05-08T23:31:12Z" level=info msg="status changed  Offline -> Online" client=eth-sepolia-backend-1
2025-05-08 23:30:57.816	time="2025-05-08T23:30:57Z" level=info msg="status changed  Online -> Offline" client=eth-sepolia-backend-1
2025-05-08 22:31:01.141	time="2025-05-08T22:31:01Z" level=info msg="status changed  Offline -> Online" client=eth-sepolia-backend-1
2025-05-08 22:30:54.813	time="2025-05-08T22:30:54Z" level=info msg="status changed  Online -> Offline" client=eth-sepolia-backend-1
2025-05-08 21:31:00.347	time="2025-05-08T21:31:00Z" level=info msg="status changed  Offline -> Online" client=eth-sepolia-backend-1
2025-05-08 21:30:51.811	time="2025-05-08T21:30:51Z" level=info msg="status changed  Online -> Offline" client=eth-sepolia-backend-1
2025-05-08 20:31:00.254	time="2025-05-08T20:31:00Z" level=info msg="status changed  Offline -> Online" client=eth-sepolia-backend-1
2025-05-08 20:30:48.808	time="2025-05-08T20:30:48Z" level=info msg="status changed  Online -> Offline" client=eth-sepolia-backend-1
2025-05-08 19:31:00.862	time="2025-05-08T19:31:00Z" level=info msg="status changed  Offline -> Online" client=eth-sepolia-backend-1
2025-05-08 19:30:45.806	time="2025-05-08T19:30:45Z" level=info msg="status changed  Online -> Offline" client=eth-sepolia-backend-1
2025-05-08 19:06:12.914	time="2025-05-08T19:06:12Z" level=info msg="status changed  Offline -> Online" client=eth-sepolia-backend-2
2025-05-08 19:06:09.889	time="2025-05-08T19:06:09Z" level=info msg="status changed  Online -> Offline" client=eth-sepolia-backend-2
2025-05-08 18:59:52.371	time="2025-05-08T18:59:52Z" level=info msg="status changed  Offline -> Online" client=eth-sepolia-backend-2
2025-05-08 18:59:49.354	time="2025-05-08T18:59:49Z" level=info msg="status changed  Online -> Offline" client=eth-sepolia-backend-2
2025-05-08 18:30:50.011	time="2025-05-08T18:30:50Z" level=info msg="status changed  Offline -> Online" client=eth-sepolia-backend-1
2025-05-08 18:30:42.804	time="2025-05-08T18:30:42Z" level=info msg="status changed  Online -> Offline" client=eth-sepolia-backend-1
2025-05-08 17:55:54.709	time="2025-05-08T17:55:54Z" level=info msg="status changed  Offline -> Online" client=eth-sepolia-backend-2
2025-05-08 17:55:51.686	time="2025-05-08T17:55:51Z" level=info msg="status changed  Online -> Offline" client=eth-sepolia-backend-2
2025-05-08 17:51:12.951	time="2025-05-08T17:51:12Z" level=info msg="status changed  Offline -> Online" client=eth-sepolia-backend-2
2025-05-08 17:51:09.925	time="2025-05-08T17:51:09Z" level=info msg="status changed  Online -> Offline" client=eth-sepolia-backend-2
2025-05-08 17:30:48.521	time="2025-05-08T17:30:48Z" level=info msg="status changed  Offline -> Online" client=eth-sepolia-backend-1
2025-05-08 17:30:39.801	time="2025-05-08T17:30:39Z" level=info msg="status changed  Online -> Offline" client=eth-sepolia-backend-1
2025-05-08 16:30:48.587	time="2025-05-08T16:30:48Z" level=info msg="status changed  Offline -> Online" client=eth-sepolia-backend-1
2025-05-08 16:30:36.799	time="2025-05-08T16:30:36Z" level=info msg="status changed  Online -> Offline" client=eth-sepolia-backend-1