🐛 fix: collect operator pod logs across restarts and rolling updates - #2848
Conversation
✅ Deploy Preview for olmv1 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
There was a problem hiding this comment.
Pull request overview
This PR improves e2e failure diagnostics by persisting olmv1-system workload logs on the KIND node filesystem so logs survive pod restarts/rollouts, then packaging those persisted logs into the support-bundle collected on failure.
Changes:
- Add a Fluent Bit Helm values file to tail container logs and write them to a hostPath-backed directory.
- Extend the e2e support-bundle spec to copy persisted Fluent Bit output from nodes into the bundle.
- Update the Makefile to deploy Fluent Bit (gated on
ARTIFACT_PATH) as part ofe2e-run-%.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
testdata/fluentbit/values.yaml |
Introduces Fluent Bit configuration/hostPath mounts to persist logs on node disks during e2e. |
test/e2e/support-bundle.yaml |
Adds a copyFromHost collector to include persisted node logs in the failure support-bundle. |
Makefile |
Adds fluentbit-% Helm deployment target and wires it into e2e-run-%. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| inputs: | | ||
| [INPUT] | ||
| Name tail | ||
| Path /var/log/containers/*_olmv1-system_*.log | ||
| multiline.parser cri |
Serial tests (HA, TLS, proxy scenarios) patch operator-controller and catalogd Deployments, triggering rolling updates that delete the original pods. The support bundle collected at the end only captures replacement pod logs, so parallel test failure logs are lost. Deploy Fluent Bit (gated on ARTIFACT_PATH) to continuously persist olmv1-system container logs to the node filesystem. A copyFromHost collector in the support bundle extracts these persisted logs into the operator-logs/ directory, preserving the complete log history across all pod generations including deleted and restarted instances. Co-Authored-By: Claude <noreply@anthropic.com>
|
/approve |
openshift-ci
Bot
added
the
approved
rashmigottipati
left a comment
rashmigottipati
left a comment
There was a problem hiding this comment.
solid improvement. thanks @pedjak
/lgtm
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: rashmigottipati, tmshort The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
fee8cc2
into
operator-framework:main
Description
The experimental-e2e CI job runs parallel tests then serial tests on the same kind cluster. Serial tests (HA, TLS, proxy scenarios) patch operator-controller and catalogd Deployments, triggering rolling updates that delete the original pods. The support bundle is collected only at the end (on failure), so it only captures replacement pod logs — parallel test failure logs are lost.
Since Kubernetes does not support retaining logs from deleted pods (kubernetes#123279), the only way to preserve them is to collect logs before the pods are removed.
This PR deploys Fluent Bit (gated on
ARTIFACT_PATH) to continuously tailolmv1-systemcontainer logs to a persistent directory on the node filesystem. AcopyFromHostcollector in the support bundle spec extracts these persisted logs into theoperator-logs/directory, preserving the complete log history across all pod generations including deleted and restarted instances.Changes
testdata/fluentbit/values.yaml— Minimal Fluent Bit Helm values: tails onlyolmv1-systempod logs, writes to/var/log/fluentbit/output/<namespace>/<pod>.<container>Makefile— Newfluentbit-%target (parallel withprometheus-%), added as dependency ofe2e-run-%test/e2e/support-bundle.yaml— AddedcopyFromHostcollector to extract persisted logs from nodesReviewer Checklist