reactor: Disable hot polling if wakeup granularity is too high #191
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We are seeing an issue where rand write performance under performs massively (100x). This is caused by the linux dio kernel thread/workqueue being starved and hence aio write completitions aren't being served in a timely manner. This doesn't happen using "default" linux settings but only if `/proc/sys/kernel/sched_wakeup_granularity_ns` or (`/sys/kernel/debug/sched/wakeup_granularity_ns` on newer kernels) is raised. Specifically this effect can be observed on RHEL-8 as the `tuned` version that ships with it sets this value to 15000000 but can reproduced on any other system by just bumping that value. This patch tries to detect this being the case and if so it will warn and disable hot polling (both `--poll-aio` and `--idle-poll-time-us`) which gives back the majority of the performance. Note because this setting has moved to debug fs on newer kernels which requires root rights to read it's actually not very likely that we will be able to detect it on these. However, RHEL8 uses an older kernel and is likely the major offender to run into this bug (we have had multiple customers run into this). Ref scylladb#2696 (cherry picked from commit 099cf61)
StephanDollberg
force-pushed
the
stephan/downstream-rhel8-252
branch
from
ed0c56d to
08f8e62
Compare
travisdowns
approved these changes
Apr 15, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Downstream scylladb#2715