fix(outputs.datadog): Prevent retry on authentication error#18163
Open
FlorentinDUBOIS wants to merge 2 commits intoinfluxdata:masterfrom
Open
fix(outputs.datadog): Prevent retry on authentication error#18163FlorentinDUBOIS wants to merge 2 commits intoinfluxdata:masterfrom
FlorentinDUBOIS wants to merge 2 commits intoinfluxdata:masterfrom
Conversation
When API tokens expire or get revoked, output plugins were retrying indefinitely, causing DDoS-like behavior against the target APIs. This issue was discovered with the warp10 plugin at Clever Cloud when token renewal failed - metrics kept retrying against endpoints returning authentication errors, creating excessive load on the Warp10 platform. Add internal.HTTPError type to distinguish retryable errors (5xx server errors, rate limits) from non-retryable errors (4xx client errors like invalid/expired tokens). Plugins now properly drop metrics on authentication failures instead of retrying forever. Updated plugins: - warp10: parse response body to determine retryability (primary fix) - azure_monitor: refactor send() to return HTTPError - datadog: return HTTPError for 4xx/5xx responses - dynatrace: return HTTPError for 4xx/5xx responses - sumologic: return HTTPError for 4xx/5xx responses - influxdb_v2: use APIError for 401/403 as non-retryable
Contributor
|
Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip. 📦 Click here to get additional PR build artifactsArtifact URLs |
mstrandboge
changed the title
telegraf-tiger
bot
added
the
plugin/output
Replace internal.HTTPError with plugin-local httpError type. 4xx client errors non-retryable, other non-2xx retryable. Add test for 403 non-retryable error. Signed-off-by: Florentin Dubois <florentin.dubois@clever.cloud>
FlorentinDUBOIS
force-pushed
the
devel/fdubois/fix/outputs/datadog
branch
from
1c45c15 to
9c3cd6d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Checklist
Related issues
relates #18119 #18118