Skip to content

ClickHouse retries on not-ready servers because of PublishNotReadyAddresses #1687

Open
@cangyin

Description

@cangyin

If a server failed to start, DNS should fail to resolve (server name does not correspond to any IP address), so as to tell the initiator ClickHouse server to avoid connecting to this server.

With PublishNotReadyAddresses=true, the initiator ClickHouse server believes a failed server should exist and makes several retries to connect it. Thus when there is any failed server, all subsequent queries will slow down for several seconds to wait for the connection retries to run out.

PublishNotReadyAddresses here

PublishNotReadyAddresses: true,

As k8s docs says:

The primary use case for setting this field is for a StatefulSet's Headless Service to propagate SRV DNS records for its Pods for the purpose of peer discovery

But a ClickHouse cluster does not need peer discovery because it is designed as multi-master architecture, where peer addresses are resolved from their hostnames before query execution (perhaps with DNS cache). An IP address of failed server does not make it better.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions