fix: log intentional Go-plugin reject responses at warn instead of error#8506
Open
lopster568 wants to merge 1 commit into
Open
fix: log intentional Go-plugin reject responses at warn instead of error#8506lopster568 wants to merge 1 commit into
lopster568 wants to merge 1 commit into
Conversation
Go-plugin middleware logged every 4xx/5xx plugin response at level=error, including intentional rejects such as 429 from rate limiting and 503 from circuit breaking, which floods severity-based alerting at scale. Map backpressure status codes (408, 418, 429, 503) to warn for the "Failed to process request with Go-plugin middleware func" log line and keep everything else, including auth failures, at error. This extends the per-error log-level convention already used in this file, where ErrResponseErrorSent carries errpack.WithLogLevel and middleware.go logs "Finished" via errpack.LogLevel. Relates to TykTechnologies#8245
|
SentinelOne CNS Hardcoded Secret Detector SentinelOne CNS is a cloud-agnostic, agentless CSPM & CWPP solution that continuously detects and prevents vulnerabilities that have the highest probability of being exploited in Azure, AWS, Google Cloud, and Kubernetes. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Go-plugin middleware logs every 4xx/5xx plugin response at
level=error, with no distinction between an unexpected failure and an intentional reject such as 429 from rate limiting or 503 from circuit breaking. This addslevelForPluginStatusingateway/mw_go_plugin.goand uses it for the "Failed to process request with Go-plugin middleware func" line inhandleErrorResponse:warnerrorThis is Option 1 from the issue, which the Visor triage bot endorsed and invited a PR for. It follows the per-error log-level convention already in this file:
ErrResponseErrorSentcarrieserrpack.WithLogLevelandgateway/middleware.gohonors it viaerrpack.LogLevel, but the spamming line fires before that wrap and hardcoded.Error().gateway/mw_jwt.gouses the same switch-then-Logshape (#8401).No config or API definition schema changes. Deployments alerting on
level=errorcounts from this line will see 408, 418, 429 and 503 move towarn, which is the point of the fix.Related Issue
Fixes #8245
Motivation and Context
At 1k QPS with 10 percent rejects that is 100 lines/sec of
level=errorfor traffic that is normal by design, which floods severity-based alerting. The JS and gRPC coprocess paths avoid this viaReturnOverrides; the Go-plugin path has no equivalent, and the only control today is the globallog_level, which also silences unrelated real errors.How This Has Been Tested
TestGoPluginMiddleware_handleErrorResponseLogLevelcallshandleErrorResponsefor 408, 418, 429 and 503 expectingwarn, and 400, 401 and 500 expectingerror, then checks the returned error still wrapsErrResponseErrorSentwith the status code unchanged. 403 is asserted onlevelForPluginStatusdirectly because that branch fires a gateway event.Locally on Go 1.25:
gofmtclean on both files,go build ./gateway/,go vet ./gateway/and the new test all pass.Screenshots (if appropriate)
N/A
Types of changes
Checklist
No documentation or go.mod changes in this PR.