Skip to content

Conversation

sebinsunny
Copy link
Contributor

@sebinsunny sebinsunny commented Oct 11, 2025

Summary

This PR implements DNS auto-resolution for load balancing in the ClickHouse sink, addressing issue #23877. The feature allows automatic discovery and rotation of ClickHouse cluster nodes through DNS lookups, enabling better high availability and load distribution when connecting to ClickHouse clusters with multiple endpoints.

Vector configuration

sinks:
  clickhouse:
    type: clickhouse
    inputs: [test]
    endpoint: "http://clickhouse-cluster.example.com:8123"
    database: "my_database"
    table: "my_table"
    auto_resolve_dns: true  # Enable DNS auto-resolution for load balancing
    # The sink will automatically resolve the DNS name and load balance across all resolved IPs

How did you test this PR?

  • Added integration tests in src/sinks/clickhouse/integration_tests.rs to verify DNS resolution functionality
  • Tested with the ClickHouse service to ensure multiple IP addresses are properly discovered, and each batch request is forwarded randomly to an IP address.
dig +short clickhouse-sebin-vector-dev-sandbox.g.aivencloud.com

Output:
34.14.24.250
34.140.127.238
34.78.18.219
  • Verified that resolved endpoints are IP addresses rather than hostnames
  • Ran existing ClickHouse sink tests to ensure no regression

Change Type

  • Bug fix
  • New feature
  • Non-functional (chore, refactoring, docs)
  • Performance

Is this a breaking change?

  • Yes
  • No

Does this PR include user facing changes?

  • Yes. Please add a changelog fragment based on our guidelines.
  • No. A maintainer will apply the no-changelog label to this PR.

References

Notes

  • Please read our Vector contributor resources.
  • Do not hesitate to use @vectordotdev/vector to reach out to us regarding this PR.
  • Some CI checks run only after we manually approve them.
    • We recommend adding a pre-push hook, please see this template.
    • Alternatively, we recommend running the following locally before pushing to the remote branch:
      • make fmt
      • make check-clippy (if there are failures it's possible some of them can be fixed with make clippy-fix)
      • make test
  • After a review is requested, please avoid force pushes to help us review incrementally.
    • Feel free to push as many commits as you want. They will be squashed into one before merging.
    • For example, you can run git merge origin master and git push.
  • If this PR introduces changes Vector dependencies (modifies Cargo.lock), please
    run make build-licenses to regenerate the license inventory and commit the changes (if any). More details here.

@sebinsunny sebinsunny requested review from a team as code owners October 11, 2025 06:22
@github-actions github-actions bot added domain: sinks Anything related to the Vector's sinks domain: external docs Anything related to Vector's external, public documentation labels Oct 11, 2025
@sebinsunny sebinsunny force-pushed the sebinsunny/clickhouse-sink-dns-support branch from 8644da3 to e5d9bb7 Compare October 11, 2025 06:37
Comment on lines +209 to +212
Automatically resolve hostnames to all available IP addresses.
When enabled, the hostname in the endpoint will be resolved to all its IP addresses,
and Vector will load balance across all resolved IPs.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Automatically resolve hostnames to all available IP addresses.
When enabled, the hostname in the endpoint will be resolved to all its IP addresses,
and Vector will load balance across all resolved IPs.
Automatically resolve hostnames to all available IP addresses.
When enabled, the hostname in the endpoint is resolved to all its IP addresses,
and Vector load balances across all resolved IPs.

Non-blocking verb tense nit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

domain: external docs Anything related to Vector's external, public documentation domain: sinks Anything related to the Vector's sinks

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ClickHouse Sink DNS Auto-Resolution for Load Balancing

2 participants