fix: use shared fsnotify watcher to avoid 'too many open files' error#236
Merged
Conversation
|
CodeReview流程已终止 |
This was referenced Feb 1, 2026
johnlanni
added a commit
to higress-group/higress
that referenced
this pull request
Feb 2, 2026
Add troubleshooting section for 'too many open files' error caused by insufficient fs.inotify.max_user_instances limit. This commonly occurs on systems with many Docker containers. Solution: Increase the limit to 8192 via sysctl. Related: higress-group/higress-standalone#236
johnlanni
added a commit
to higress-group/higress
that referenced
this pull request
Feb 2, 2026
Move detailed troubleshooting content from SKILL.md to references/TROUBLESHOOTING.md to keep the main skill file concise. Changes: - Create references/TROUBLESHOOTING.md with all troubleshooting content - Replace Troubleshooting section in SKILL.md with summary + link - Add inotify max_user_instances issue and solution Related: higress-group/higress-standalone#236
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The API server crashes with panic:
too many open fileswhen creating REST storage for resources.Root cause: Each resource type (ConfigMap, Secret, Ingress, etc.) creates its own
fsnotify.Watcherinstance inNewFileREST(), which consumes file descriptors. When the number of resource types exceeds the system limit, the application crashes.Solution
Use a shared
fsnotify.Watcheracross all fileREST instances:completedConfig.New()for File storage modeNewFileREST()callsDestroy()to not close the watcher (managed by server lifecycle)Changes
src/apiserver/pkg/apiserver/apiserver.go: Create and pass shared watchersrc/apiserver/pkg/registry/file_rest.go: Accept shared watcher parameter, remove per-instance watcher creationTesting
Before: Application crashes with "too many open files" when registering multiple resources
After: All resources share one watcher, file descriptor usage reduced significantly
Fixes the panic reported in production environments.