Bug: table migration failure doesn't return error

[dentiny]

Hi team, we're deploying clickhouse in kubernetes cluster with this project. One thing we found weird is, in rare cases we suffer error messages like There is no table xxx on server, which indicates a missing schema population issue.

When checking the source code, I found on table migration failure, we only log error messages but not return error.
IIUC, during reconciliation, if we ignore the migration failure, certain host status will be transitioned from "Requested" to "Success", then serve traffic and suffer schema errors.

[alex-zaitsev]

@dentiny , thank you for PR, it makes sense, but it is not clear how to fix replica if schema creation fails.

Could you show the error message, what was the problem why table has not been properly created?

[dentiny]

Hi @alex-zaitsev , thanks for the quick response! I think the thing is

• Schema migration can fail (discovery timeout, CREATE error, etc.), but those errors were logged and discarded instead of stopping the host reconcile
• But the operator still marked the host as tables-created and continued to include it in remote_servers as if migration succeeded. Distributed queries then routed traffic to a shard that was up but had no local tables, producing “table does not exist” / Code 279.
• Because the host was already recorded as schema-complete, later reconciles often skipped migration entirely, so the empty schema could stick until manual intervention.