fix(e2e): enable persistent journal to capture post-reboot logs#1465
fix(e2e): enable persistent journal to capture post-reboot logs#1465qinqon wants to merge 1 commit intonmstate:mainfrom
Conversation
The kubevirtci VMs use volatile journal by default, so logs are lost when a node reboots during tests. This makes it impossible to debug failures in reboot tests (e.g. cni0 not recreated after OVS bridge removal). Create /var/log/journal on each node during cluster-up so systemd-journald (Storage=auto) persists logs across reboots. Signed-off-by: Enrique Llorente <ellorent@redhat.com> Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
kubevirt-bot
added
release-note-none
kubevirt-bot
added
the
dco-signoff: yes
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses a debugging challenge in e2e tests by ensuring that systemd journal logs persist across node reboots. Previously, logs were lost, making it difficult to diagnose issues in tests that involve node restarts. The change implements a mechanism to retain these crucial logs, thereby enhancing the ability to troubleshoot test failures effectively. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
The pull request successfully enables persistent systemd journal on kubevirtci nodes by creating the /var/log/journal directory and restarting the systemd-journald service. This change directly addresses the issue of lost logs after node reboots during e2e tests, significantly improving debugging capabilities. A minor efficiency improvement could be considered for the service restart operation to avoid unnecessary interruptions.
| ./cluster/cli.sh ssh ${node} -- sudo systemctl restart NetworkManager | ||
| # Enable persistent journal so logs survive node reboots | ||
| ./cluster/cli.sh ssh ${node} -- sudo mkdir -p /var/log/journal | ||
| ./cluster/cli.sh ssh ${node} -- sudo systemctl restart systemd-journald |
There was a problem hiding this comment.
To improve efficiency and avoid unnecessary service interruptions, consider restarting systemd-journald only if it's not already configured for persistent storage. While mkdir -p /var/log/journal ensures the directory exists, systemd-journald might already be running with persistent storage enabled from a previous run. An unconditional restart on every node in the loop could lead to minor performance overhead and temporary logging unavailability. A conditional restart would be more robust.
| ./cluster/cli.sh ssh ${node} -- sudo systemctl restart systemd-journald | |
| ./cluster/cli.sh ssh ${node} -- sudo bash -c 'if ! journalctl --disk-usage | grep -q "Persistent: yes"; then systemctl restart systemd-journald; fi' |
There was a problem hiding this comment.
This runs once during cluster-up on freshly provisioned VMs, so there's no "previous run" scenario — the nodes are always new. The restart takes milliseconds and the script already restarts heavier services (NetworkManager, openvswitch) unconditionally in the same loop. Adding a conditional check would be more complex for no practical benefit here.
|
/lgtm |
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: mkowalski The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
kubevirt-bot
added
the
approved
|
@qinqon: The following test failed, say
DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
3 participants
/kind bug
What this PR does / why we need it:
Enables persistent systemd journal on kubevirtci nodes so that logs survive node reboots during e2e tests.
Currently the VMs use volatile journal (
/run/log/journal), which means all logs are lost when a test reboots a node. This makes it impossible to debug failures in reboot tests — for example, in build 2036261859606138880 theshould keep the default IP address after node reboottest failed becausecni0wasn't recreated after an OVS bridge was removed, but the post-reboot flannel/kubelet logs were not available to diagnose why.The fix creates
/var/log/journalon each node duringcluster-up. Since systemd-journald defaults toStorage=auto, this is sufficient to enable persistent storage across reboots.Special notes for your reviewer:
No functional test changes — this only affects log collection for debugging.
Release note: