Idea/Feature Request - Refactor role with basic settings + free-form XML config

[AntonFriberg]

Hi!

This is just an idea that I would want to discuss with the maintainer and community of this project. I have written a couple of pull-requests due to needs that I have in my production environment that was not exposed in this role. I have noticed that it is quite cumbersome for this role to maintain all the small configuration options that ClickHouse allows and at the same time I feel limited if I am not able to do the changes I would like.

It would not be that difficult to simply deploy my own XML config files next to this roles generated files but that results in added complexity due to having configuration in multiple places. So instead I suggest a different approach.

Let this role handle basic configuration that pretty much everyone want to change but allow additional configuration that results in separate XML config files. Since the free-flow config options are just multi-line strings we can utilize ansible templating as normal which is very powerful.

Is this something that could be valuable for this role? Do you want me to provide a PR? Let us discuss!

It would look something like this in an Ansible Inventory file (yaml in my case).

---
all:
  hosts: # Host aliases
    clickhouse[01:04].domain.com:
      ansible_port: 22
      ansible_user: "clickhouse"
    clickhouse01:
      ansible_host: "clickhouse01.domain.com"
      # node specific variables
      clickhouse_macros: # Node specific variables
        cluster: "cluster-name"
        shard: "01"
        replica: "clickhouse01"
    clickhouse02:
      ansible_host: "clickhouse02.domain.com"
      clickhouse_macros:
        cluster: "cluster-name"
        shard: "01"
        replica: "clickhouse02"
    clickhouse03:
      ansible_host: "clickhouse03.domain.com"
      clickhouse_macros:
        cluster: "cluster-name"
        shard: "02"
        replica: "clickhouse03"
    clickhouse04:
      ansible_host: "clickhouse04.domain.com"
      clickhouse_macros:
        cluster: "cluster-name"
        shard: "02"
        replica: "clickhouse04"

  vars: # Variables for all groups

    # Very important config is on first level as expected
    clickhouse_version: "22.2.2.1"
    clickhouse_http_port: 8123
    clickhouse_tcp_port: 9000
    clickhouse_interserver_http: 9009
    clickhouse_listen_host:
      - "::"

    # Stable configuration options that most users want is also available
    clickhouse_config:
      max_connections: 4096
      keep_alive_timeout: 3
      max_concurrent_queries: 100
      uncompressed_cache_size: 8589934592
      mark_cache_size: 5368709120
      builtin_dictionaries_reload_interval: 3600
      dictionaries_lazy_load: true
      max_session_timeout: 3600
      default_session_timeout: 60
      default_profile: "default"
      default_database: "some_database"

    # Expose ability to define free-form configuration
    # These will be added as files under config.d/
    clickhouse_custom_config:
      - file_name: "clickhouse_ldap_servers.xml"
        content: |
          <clickhouse>
              <ldap_servers>
                  <company_ldap>
                      <host>ldap.company.com</host>
                      <port>636</port>
                      <bind_dn>COMPANYDOMAIN\\{user_name}</bind_dn>
                      <enable_tls>yes</enable_tls>
                      <user_dn_detection>
                          <base>OU=Company,DC=company,DC=com</base>
                          <search_filter>(&amp;(objectClass=user)(sAMAccountName={user_name}))</search_filter>
                      </user_dn_detection>
                  </company_ldap>
              </ldap_servers>
          </clickhouse>
      - file_name: "clickhouse_s3_endpoints.xml"
        content: |
          <clickhouse>
            <s3>
                <aws-datalake>
                    <endpoint>https://bucket-name.s3.us-east-1.amazonaws.com</endpoint>
                    <access_key_id>some_id_string</access_key_id>
                    <secret_access_key>some_secret_string</secret_access_key>
                    <region>us-east-1</region>
                </aws-datalake>
            </s3>
          </clickhouse>