[operator] healthchecks stuck in RUNNING state

[fruch]

during k8s functional test, we something get into a sitiuation, the health checks are not continuing

< t:2022-07-12 08:33:01,259 f:cli.py          l:1061 c:sdcm.mgmt.cli        p:DEBUG > Issuing: 'sctool tasks -c db77d799-dfc0-4ff4-91f5-81d792ccfa49'
< t:2022-07-12 08:33:01,259 f:remote_base.py  l:520  c:KubernetesCmdRunner  p:DEBUG > Running command "sctool tasks -c db77d799-dfc0-4ff4-91f5-81d792ccfa49"...
< t:2022-07-12 08:33:01,372 f:base.py         l:222  c:KubernetesCmdRunner  p:DEBUG > +------------------------+-------------+--------+----------+---------+-------+--------------+------------+---------+------+
< t:2022-07-12 08:33:01,372 f:base.py         l:222  c:KubernetesCmdRunner  p:DEBUG > | Task                   | Schedule    | Window | Timezone | Success | Error | Last Success | Last Error | Status  | Next |
< t:2022-07-12 08:33:01,372 f:base.py         l:222  c:KubernetesCmdRunner  p:DEBUG > +------------------------+-------------+--------+----------+---------+-------+--------------+------------+---------+------+
< t:2022-07-12 08:33:01,372 f:base.py         l:222  c:KubernetesCmdRunner  p:DEBUG > | healthcheck/rest       | @every 1m0s |        | UTC      | 12      | 2     | 9m50s ago    | 8m50s ago  | RUNNING |      |
< t:2022-07-12 08:33:01,372 f:base.py         l:222  c:KubernetesCmdRunner  p:DEBUG > | healthcheck/alternator | @every 15s  |        | UTC      | 53      | 6     | 9m5s ago     | 7m26s ago  | RUNNING |      |
< t:2022-07-12 08:33:01,372 f:base.py         l:222  c:KubernetesCmdRunner  p:DEBUG > | healthcheck/cql        | @every 15s  |        | UTC      | 53      | 6     | 9m5s ago     | 3m5s ago   | RUNNING |      |
< t:2022-07-12 08:33:01,372 f:base.py         l:222  c:KubernetesCmdRunner  p:DEBUG > +------------------------+-------------+--------+----------+---------+-------+--------------+------------+---------+------+

and after 10 minutes, all health check in running state, and not done as we expect:

< t:2022-07-12 08:43:14,095 f:base.py         l:142  c:KubernetesCmdRunner  p:DEBUG > Command "sctool tasks -c db77d799-dfc0-4ff4-91f5-81d792ccfa49" finished with status 0
< t:2022-07-12 08:43:14,095 f:cli.py          l:1064 c:sdcm.mgmt.cli        p:DEBUG > sctool output: +------------------------+-------------+--------+----------+---------+-------+--------------+------------+---------+------+
< t:2022-07-12 08:43:14,095 f:cli.py          l:1064 c:sdcm.mgmt.cli        p:DEBUG > | Task                   | Schedule    | Window | Timezone | Success | Error | Last Success | Last Error | Status  | Next |
< t:2022-07-12 08:43:14,095 f:cli.py          l:1064 c:sdcm.mgmt.cli        p:DEBUG > +------------------------+-------------+--------+----------+---------+-------+--------------+------------+---------+------+
< t:2022-07-12 08:43:14,095 f:cli.py          l:1064 c:sdcm.mgmt.cli        p:DEBUG > | healthcheck/rest       | @every 1m0s |        | UTC      | 12      | 2     | 20m3s ago    | 19m3s ago  | RUNNING |      |
< t:2022-07-12 08:43:14,095 f:cli.py          l:1064 c:sdcm.mgmt.cli        p:DEBUG > | healthcheck/alternator | @every 15s  |        | UTC      | 53      | 6     | 19m18s ago   | 17m39s ago | RUNNING |      |
< t:2022-07-12 08:43:14,095 f:cli.py          l:1064 c:sdcm.mgmt.cli        p:DEBUG > | healthcheck/cql        | @every 15s  |        | UTC      | 53      | 6     | 19m18s ago   | 13m18s ago | RUNNING |      |
< t:2022-07-12 08:43:14,095 f:cli.py          l:1064 c:sdcm.mgmt.cli        p:DEBUG > +------------------------+-------------+--------+----------+---------+-------+--------------+------------+---------+------+

Logs

• db-cluster - https://cloudius-jenkins-test.s3.amazonaws.com/ff15b5be-75ec-452b-a82f-7934874bf220/20220712_092944/db-cluster-ff15b5be.tar.gz
• kubernetes - https://cloudius-jenkins-test.s3.amazonaws.com/ff15b5be-75ec-452b-a82f-7934874bf220/20220712_092944/kubernetes-ff15b5be.tar.gz
• sct - https://cloudius-jenkins-test.s3.amazonaws.com/ff15b5be-75ec-452b-a82f-7934874bf220/20220712_092944/sct-runner-ff15b5be.tar.gz

[fruch]

Seem like this wait for healthcheck to end was added cause of an issue scylladb/scylla-cluster-tests#4699

the code added for waiting for healthcheck to be done:
scylladb/scylla-cluster-tests@e81df14

[fruch]

added a sctool status to see the cluster status at that time:

< t:2022-07-12 20:46:36,355 f:base.py         l:222  c:KubernetesCmdRunner  p:DEBUG > Error: get cluster "a539e0fe-2b37-4809-8249-5efb182ee17c" status: get client: after 30s: context deadline exceeded
< t:2022-07-12 20:46:36,355 f:base.py         l:222  c:KubernetesCmdRunner  p:DEBUG > Trace ID: 4FVe0A3OS6iqntbHnwRNfQ (grep in scylla-manager logs)

seems like the whole scylla cluster node were replaced and got new addresses, does the operator need to update the manager somehow ? @scylladb/team-operator ?

[fruch]

@zimnx mentioned in slack:

PodIPs are ephemeral, that's why we register clusters in manager using stable DNS name. Looks like the IP address is cached within manager

[fruch]

seems like it's mentioned in:
https://github.com/scylladb/scylla-manager-enterprise/issues/1597

but I don;t think any of what mentioned there, ever happened