[release-5.0] OCPBUGS-121354: feat: add cert-watcher DaemonSet to restart etcd on CA bundle rotation - #1702
Conversation
In TNF deployments, etcd runs under Pacemaker/podman rather than as a Kubernetes static pod. When the CA bundle is rotated, etcd does not automatically reload certificates. The kube-apiserver, which presents a client certificate signed by the new CA, gets rejected by etcd with "tls: uknown certificate autoritty", causing API server outages. The cert-watcher DaemonSet runs on each control plane node and monitors the CA bundle directory for changes using fsnotify with a 1-minute fallback poll. Whan a change is detected, the watcher: 1. Verifies the entire etcd cluster is healthy (serializes restarts across the 2-node cluster to avoid simultatneuous quorum loss) 2. Sets 'restart_no_leave' on the LOCAL node only via crm_attribute (prevents the OCF agent from running 'force_new_cluster' on restart) 3. Restart etcd via 'podman restart etcd' (SIGTERM preserves cluster membership, unlike 'pcs resource reestart' which is a no-op while unmanaged) 4. Waits for etcd to become healthy again (up to 5 minutes) 5. Cleans up any Pacemaker failure state caused by restart
|
Pipeline controller notification For optional jobs, comment This repository is configured in: LGTM mode |
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository: openshift/coderabbit/.coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
@openshift-cherrypick-robot: Jira Issue OCPBUGS-104851 has been cloned as Jira Issue OCPBUGS-121354. Will retitle bug to link to clone. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci
Bot
changed the title
openshift-ci-robot
added
jira/severity-critical
|
@openshift-cherrypick-robot: This pull request references Jira Issue OCPBUGS-121354, which is invalid:
Comment The bug has been updated to refer to the pull request using the external bug tracker. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci-robot
added
the
jira/invalid-bug
|
/jira refresh |
openshift-ci-robot
added
jira/valid-bug
|
@fracappa: This pull request references Jira Issue OCPBUGS-121354, which is valid. The bug has been moved to the POST state. 7 validation(s) were run on this bug
No GitHub users were found matching the public email listed for the QA contact in Jira (dhensel@redhat.com), skipping review request. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@openshift-cherrypick-robot: all tests passed! Full PR test history. Your PR dashboard. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/lgtm |
|
Scheduling required tests: |
|
/payload-job periodic-ci-openshift-release-main-nightly-5.0-e2e-metal-ovn-two-node-fencing-upgrade periodic-ci-openshift-release-main-nightly-5.0-e2e-metal-ovn-two-node-fencing-recovery |
|
@lucaconsalvi: trigger 4 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/c60f1e60-aab8-11f1-98cb-042b8abe9875-0 |
|
/payload-job-with-prs periodic-ci-openshift-release-main-nightly-5.0-e2e-metal-ovn-two-node-fencing-recovery openshift/origin#31530 openshift/origin#31597 |
|
@lucaconsalvi: trigger 3 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/dcac9bc0-aab8-11f1-907d-246587705043-0 |
|
/payload-job periodic-ci-openshift-release-main-nightly-5.0-e2e-metal-ovn-two-node-fencing-etcd-certrotation |
|
@lucaconsalvi: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/fcf5dc90-aac0-11f1-95b4-96a08cbbc2d3-0 |
4 participants
This is an automated cherry-pick of #1675
/assign fracappa