WIP / Experiment: reference counting directory FDs to keep them alive longer #169
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Done, it now reserves |
bertschinger
force-pushed
the
dir_refcounting
branch
from
7a5421b to
9f354d3
Compare
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #169 +/- ##
==========================================
- Coverage 91.08% 90.96% -0.13%
==========================================
Files 57 57
Lines 8394 8454 +60
Branches 1114 1124 +10
==========================================
+ Hits 7646 7690 +44
- Misses 467 477 +10
- Partials 281 287 +6 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
bertschinger
force-pushed
the
dir_refcounting
branch
from
9f354d3 to
9e6086c
Compare
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
bertschinger
force-pushed
the
dir_refcounting
branch
3 times, most recently
from
3378a3d to
8111a08
Compare
Comment on lines
+101
to
+103
| // fprintf(stderr, "Warning: system may not allow enough open files for the number of requested threads.\n"); | ||
| // fprintf(stderr, "Max number of open files: %llu; Number of threads requested: %llu\n", rl.rlim_cur, nthreads); |
Check notice
Code scanning / CodeQL
Commented-out code Note
bertschinger
force-pushed
the
dir_refcounting
branch
3 times, most recently
from
b85370b to
1985be7
Compare
calccrypto
reviewed
Mar 24, 2025
calccrypto
reviewed
Mar 24, 2025
calccrypto
reviewed
Mar 24, 2025
calccrypto
reviewed
Mar 24, 2025
calccrypto
reviewed
Mar 24, 2025
| struct rlimit rl; | ||
| int res = getrlimit(RLIMIT_NOFILE, &rl); | ||
| if (res) { | ||
| fprintf(stderr, "Warning: could not get open file limit: %s\n", strerror(errno)); |
calccrypto
force-pushed
the
main
branch
2 times, most recently
from
fd865c3 to
5ab578d
Compare
this is needed when the queues hold items that need additional work to free above just calling free(). so that they can clean up everything on exit properly.
Since keeping directory FDs alive in order to open child directories with relative paths incurs the risk of exhausting available file descriptors, this tracks the maximum number of FDs that should be allocated to long-living directory handles. When that maximum is reached, directory handles will no longer have their lifetime extended, until the limit goes back down again.
bertschinger
force-pushed
the
dir_refcounting
branch
from
1985be7 to
cd2700f
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is a work-in-progress, experimental branch that tests refcounting dir FDs with the goal of keeping them alive while there are still children waiting to be processed.
The idea is to see if this helps the cache behavior on Lustre with the Lustre inode cache disabled.
This also enables using openat() in the processdir() functions in order to reduce pathname resolution overhead.