[ANSIENG-5684] | connectors deployment fix #2353
Conversation
|
Uploading logs-multicluster.log… |
This doesn't seem correct |
I've updated the description |
| - "{{kafka_connect_cert_path}}" | ||
| - "{{kafka_connect_key_path}}" | ||
| when: (ssl_provided_keystore_and_truststore | bool) | ||
| tags: deploy_connectors |
There was a problem hiding this comment.
why is this tag included here?
There was a problem hiding this comment.
This would be a helpful feature for customer to be able to skip deploying connectors during their upgrads if they want to.
There was a problem hiding this comment.
does this work ?
when we give --tags deploy_connectors does it go inside kafka connect roles tasks main.yml ?
| hosts: kafka_connect | ||
| gather_facts: false | ||
| tags: kafka_connect | ||
| environment: "{{ proxy_env }}" |
There was a problem hiding this comment.
why do we give this proxy_env here?
There was a problem hiding this comment.
customer can set proxy env vars, hence added it here as well
| - name: Filter subgroups for rolling deployment strategy | ||
| set_fact: | ||
| filtered_subgroups: "{{ [parent_kafka_connect_cluster_group] if ((kafka_connect_deployment_strategy == 'rolling' or kafka_connect_deployment_strategy == 'serial') and parent_kafka_connect_cluster_group is defined and parent_kafka_connect_cluster_group in subgroups) else subgroups }}" | ||
| when: subgroups is defined |
There was a problem hiding this comment.
is it possible for subgroups to be not defined ?
There was a problem hiding this comment.
yes, In case of parallel run and no children structure, there are no subgroups
| when: hostvars[groups[item][0]].kafka_connect_connectors is defined | ||
| delegate_to: "{{ groups[item][0] }}" | ||
| loop: "{{subgroups}}" | ||
| loop: "{{ filtered_subgroups | default(subgroups) }}" |
There was a problem hiding this comment.
is it possible for filtered_subgroups to be not defined ?
Description
Problem:
With deployment_strategy: rolling in multi-cluster Kafka Connect setups, each host attempted to deploy connectors to all clusters, causing failures when accessing files on remote cluster nodes (e.g., [Errno 2] No such file or directory).
Fixes # (issue)
Added filtering logic so each host only deploys connectors for its own cluster during rolling/serial deployments.
Fixed the subgroups difference filter to use a list instead of a string.
Changes:
Behavior:
Rolling/Serial: Each host deploys connectors only for its own cluster
Parallel deployment:
All hosts run simultaneously, but each host only processes connectors for its own cluster
With run_once: true, connectors are deployed once per cluster (not per host)
Works as before; connectors deploy via the single-cluster task
Type of change
How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration
Checklist: