Add exponential backoff and log suppression for scheduler disconnection#6
Merged
Merged
Conversation
44ea3ba to
e18eab0
Compare
When the scheduler disconnects or becomes unavailable, the executor poll loop previously logged a WARN message every ~100ms, spamming logs during graceful shutdown scenarios. Changes: - Add exponential backoff (100ms to 30s) when scheduler connection fails - Reduce log level from WARN to DEBUG after 5 consecutive failures - Log restoration message when connection is re-established - Prevents log spam when scheduler shuts down before executor
e18eab0 to
b9b273d
Compare
peasee
approved these changes
Jan 13, 2026
phillipleblanc
added a commit
to spiceai/spiceai
that referenced
this pull request
Jan 13, 2026
Updates datafusion-ballista fork to include fix for executor log spam when scheduler disconnects before executor. The executor poll loop now uses exponential backoff (100ms to 30s) and reduces log level from WARN to DEBUG after 5 consecutive failures. See: spiceai/datafusion-ballista#6
github-merge-queue Bot
pushed a commit
to spiceai/spiceai
that referenced
this pull request
Jan 13, 2026
#8905) Updates datafusion-ballista fork to include fix for executor log spam when scheduler disconnects before executor. The executor poll loop now uses exponential backoff (100ms to 30s) and reduces log level from WARN to DEBUG after 5 consecutive failures. See: spiceai/datafusion-ballista#6
lukekim
pushed a commit
to spiceai/spiceai
that referenced
this pull request
Jan 15, 2026
#8905) Updates datafusion-ballista fork to include fix for executor log spam when scheduler disconnects before executor. The executor poll loop now uses exponential backoff (100ms to 30s) and reduces log level from WARN to DEBUG after 5 consecutive failures. See: spiceai/datafusion-ballista#6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When the scheduler disconnects or becomes unavailable, the executor poll loop previously logged a WARN message every ~100ms, spamming logs during graceful shutdown scenarios (e.g., when shutting down the scheduler before the executor).
Before:
After:
Changes