fsnotify: Fix checkpoint failure for inotify watches on overlayfs#3048
Closed
ankimaha-sys wants to merge 1 commit into
Closed
fsnotify: Fix checkpoint failure for inotify watches on overlayfs#3048ankimaha-sys wants to merge 1 commit into
ankimaha-sys wants to merge 1 commit into
Conversation
Member
|
Looks like what I did a couple of days ago #3043 😉 |
Author
|
Thanks for the heads up @adrianreber! I arrived at the same fix independently — good to know we're aligned on the approach. I've updated this PR to be more complete:
Happy to close this in favour of #3043 if you prefer, or if there's anything useful here you'd like to cherry-pick, feel free. Either way — glad this is getting fixed! |
51c6bb7 to
47bbe0a
Compare
When checkpointing a container with inotify watches on files backed by overlayfs, CRIU fails with: fsnotify: Handle 0x34c:0x1961af5a cannot be opened Error (criu/fsnotify.c:284): fsnotify: Can't dump that handle Root cause: In alloc_openable(), suitable_mount_found is only set after open_by_handle_at() succeeds. On overlayfs, the mount with matching s_dev is found, but open_by_handle_at() fails because overlayfs does not reliably support file handle decoding. This causes check_open_handle() to take the fault path where irmap_lookup() also fails since its hint directories don't cover container filesystem paths. Fix this by: 1. Setting suitable_mount_found as soon as a mount with matching s_dev is found, before attempting open_by_handle_at(). This correctly reports ERR_NO_PATH_IN_MOUNT to the caller. 2. Adding overlayfs-specific path resolution in check_open_handle() that scans the overlay mount tree for the matching inode using a new scan_dir_for_inode() helper (depth-limited to 64, warns after 10,000 entries scanned). 3. Adding a ZDTM test (inotify_overlayfs) that mounts an overlay, creates an inotify watch, checkpoints/restores, and verifies events still work. Fixes: https://issues.redhat.com/browse/OCPBUGS-87023 Signed-off-by: Ankit Mahajan <ankimaha@redhat.com> Co-authored-by: Cursor <cursoragent@cursor.com>
47bbe0a to
59d5b4d
Compare
Member
|
@ankimaha-sys Thanks for working on this! I went ahead and merged #3043 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix CRIU checkpoint failure when dumping inotify watches on files backed by overlayfs, as seen on OpenShift 4.18 with .NET 8 container workloads.
Error before fix:
Root cause: In
alloc_openable(),suitable_mount_foundis only set afteropen_by_handle_at()succeeds. On overlayfs, the mount with matchings_devis found, butopen_by_handle_at()fails because overlayfs does not reliably support file handle decoding (depends on kernel config andnfs_exportmount option). Sincesuitable_mount_foundstays 0, the function returnsERR_NO_MOUNT, sendingcheck_open_handle()to the fault path whereirmap_lookup()also fails because its hint directories don't cover container filesystem paths.Changes
Move
suitable_mount_found = 1inalloc_openable()to before theopen_by_handle_at()attempt, so the mount is correctly reported as found even when handle decoding fails.Add overlayfs-specific path resolution in
check_open_handle(): WhenERR_NO_PATH_IN_MOUNTis returned and the mount is overlayfs, scan the overlay mount tree for the matching inode using a newscan_dir_for_inode()helper function.Add
scan_dir_for_inode()helper: Recursive directory scanner that finds a file by(s_dev, i_ino)within a mount tree. Respects mount boundaries (won't cross devices), limits recursion depth to 32, and doesn't follow symlinks.Test plan
criu dumpor kubelet checkpoint APICan't dump that handleerrormake test/ zdtm fsnotify tests)Fixes: https://issues.redhat.com/browse/OCPBUGS-87023
Made with Cursor