Skip to content

reloader: watch parent directory to survive atomic config replace#8874

Open
Goutham-Annem wants to merge 1 commit into
thanos-io:mainfrom
Goutham-Annem:reloader-watch-parent-dir-for-atomic-replace
Open

reloader: watch parent directory to survive atomic config replace#8874
Goutham-Annem wants to merge 1 commit into
thanos-io:mainfrom
Goutham-Annem:reloader-watch-parent-dir-for-atomic-replace

Conversation

@Goutham-Annem

Copy link
Copy Markdown
  • I added CHANGELOG entry for this change.
  • Change is not relevant to the end user.

Changes

  • watcher.addFile watched the config file's path directly via fsnotify.Watcher.Add(name). On Linux (inotify), this watches the file's inode. When the file is replaced atomically (temp-file-then-rename — how ConfigMap-mounted files and many editors update files in place), the original inode is gone and the watch dies with no further events, exactly as documented in fsnotify's README (linked from reloader: config file stops being watched in case of atomic updates #8127).
  • Changed addFile to watch the file's parent directory instead, mirroring the existing addDirectory (already used for CfgDirs/WatchedDirs). The existing event filter in watcher.run already matches on filepath.Dir(event.Name), so no other changes were needed.

Verification

  • Added TestReloader_ConfigApplyAtomicReplace, which replaces the watched config file atomically (write-to-temp + rename) twice in a row and asserts a reload follows each replacement.
    • A single atomic replace isn't enough to distinguish correct behavior from the bug: on Linux, the dying file-level watch still emits one trailing IN_IGNORED/remove-ish event before going dead, which is enough to trigger one reload even with the bug present. Only the second replacement reliably tells a directory-level watch (survives) apart from a file-level watch (now silently dead).
    • Note on local verification: I ran this locally on macOS, where fsnotify's kqueue backend has its own built-in compatibility shim (sendCreateIfNew in backend_kqueue.go) that re-establishes a file-level watch when it detects a new file reappearing at the same path — so the bug doesn't reproduce on macOS regardless of this fix. I confirmed by reading both backends (backend_inotify.go has no equivalent re-add logic) that this is Linux-specific, which matches the project's actual CI platform (ubuntu-latest) and the original reporter's environment.
  • go test ./pkg/reloader/...: all tests pass except the two already self-marked t.Skip("Flaky") (TestReloader_ConfigDirApply, TestReloader_ConfigDirApplyBasedOnWatchInterval) and TestReloader_DirectoriesApply, which I confirmed fails identically on main without this change (pre-existing flakiness, already tracked by Flaky TestReloader_ConfigDirApplyBasedOnWatchInterval #8114/Flaky TestCompactWithStoreGateway #8115 with open PRs against it) — unrelated to this change.
  • make go-lint passes with 0 issues.

Closes #8127

addFile watched the config file's inode directly via fsnotify. When the
file is replaced atomically (temp-file-then-rename, which is how
ConfigMap-mounted files and many editors update files in place), the
watch goes dead with no further events once the original inode is
replaced, matching the behavior documented in fsnotify's own README.

Watch the parent directory instead, mirroring how addDirectory already
handles the cfgDirs/watchedDirs case, so the watch survives replacement.

Fixes thanos-io#8127

Signed-off-by: Goutham Annem <gouthemannem@gmail.com>
@Goutham-Annem Goutham-Annem force-pushed the reloader-watch-parent-dir-for-atomic-replace branch from ef14cf0 to 1ea8c89 Compare July 4, 2026 08:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

reloader: config file stops being watched in case of atomic updates

1 participant