etcd certs regenerate on every playbook run, etcd never restarted — in-memory and on-disk TLS context drifts #13353

What happened?

Every Ansible playbook run regenerates all etcd TLS certificates, but etcd is never restarted after regeneration. This causes in-memory TLS context and on-disk certificates to diverge on every run. If etcd is later restarted for any reason, it loads newly-generated certificates that peers have never seen, causing TLS handshake failures across the cluster.

Three compounding issues are responsible:

Bug 1 — force_etcd_cert_refresh: true forces regeneration every run
roles/etcd_defaults/defaults/main.yml:18

force_etcd_cert_refresh: true

This default causes gen_certs to always evaluate true at roles/etcd/tasks/check_certs.yml:46 because force_etcd_cert_refresh short-circuits the or condition. The cert-existence check is never evaluated. This is not overridden in the sample inventory.

This was introduced in PR #7219 (commit aad7884) as a band-aid to close issue #6673. The commit message says it was intended to "maintain existing functionality" — meaning certs were already regenerating every run before this flag existed. The real root cause was never fixed.

Bug 2 — Handler timing prevents etcd restart
roles/etcd/tasks/main.yml:81-87:

- name: Restart etcd if certs changed
  command: /bin/true
  notify: Restart etcd
  when:
    - (etcd in group_names)
    - etcd_cluster_setup
    - etcd_secret_changed    # ← checked BEFORE handler fires

roles/etcd/handlers/main.yml:52-54:

- name: Set etcd_secret_changed
  set_fact:
    etcd_secret_changed: true    # ← fires at END of play, too late

gen_certs_script.yml notifies Set etcd_secret_changed at lines 47, 59, 83, 115.

Ansible handlers fire at end-of-play, not when notified. The when condition evaluates etcd_secret_changed: false (initialized at check_certs.yml:15) during the play, before the handler sets it to true. Restart etcd is never notified.

The cross-role usage of etcd_secret_changed in roles/kubernetes/control-plane/tasks/pre-upgrade.yml:9 works correctly because that runs in a separate play — handlers fire between plays. The fix must preserve this.

Bug 3 (minor) — Wrong handler name for etcd-events
roles/etcd/tasks/main.yml:91:

  notify: Restart etcd    # ← should be "Restart etcd-events"

Copy-paste bug. The "Restart etcd-events if certs changed" task notifies the regular etcd handler instead of the etcd-events handler.

Impact chain:
Every playbook run → new certs written to disk → in-memory TLS context unchanged → drift begins. On manual etcd restart → new on-disk certs loaded → peers reject TLS handshake → cluster failure.

What did you expect to happen?

1. Certificates only regenerated when missing or manually forced
2. etcd restarted when certificates change
3. In-memory and on-disk TLS context always match

How can we reproduce it?

1. Deploy a cluster with Kubespray using default settings
2. Run ansible-playbook cluster.yml a second time
3. Observe cert generation tasks (make-ssl-etcd.sh) execute again
4. Observe etcd is NOT restarted
5. Manually restart an etcd member → TLS handshake errors from peers

Version of Kubespray:

git rev-parse --short HEAD

Network plugin used:
calico (or whichever was tested)

Anything else we need to know?

• Code quality note: In roles/etcd/tasks/check_certs.yml:5, the find module's patterns parameter is passed as a comma-separated string instead of a YAML list. While Ansible handles this gracefully at runtime, the documented API expects a list, and it should be updated for idiomatic correctness.
• Ansible fact caching: ansible.cfg uses fact_caching = jsonfile with 86400s timeout, so set_fact values persist across plays. This ensures etcd_secret_changed propagates correctly from the etcd play to the control-plane play.
• Related issues: force_etcd_cert_refresh variable ignored in condition #11931 (closed by stale bot without maintainer review — user reported force_etcd_cert_refresh: false not working), etcd nodes' certificates are always created (even if they already exist) #6673 (original report of cert regeneration, closed by band-aid PR Updated etcd cert check tasks to detect when new cert gen is required #7219)