Skip to content

fix: log intentional Go-plugin reject responses at warn instead of error#8506

Open
lopster568 wants to merge 1 commit into
TykTechnologies:masterfrom
lopster568:fix-plugin-reject-error-log-level
Open

fix: log intentional Go-plugin reject responses at warn instead of error#8506
lopster568 wants to merge 1 commit into
TykTechnologies:masterfrom
lopster568:fix-plugin-reject-error-log-level

Conversation

@lopster568

Copy link
Copy Markdown

Description

Go-plugin middleware logs every 4xx/5xx plugin response at level=error, with no distinction between an unexpected failure and an intentional reject such as 429 from rate limiting or 503 from circuit breaking. This adds levelForPluginStatus in gateway/mw_go_plugin.go and uses it for the "Failed to process request with Go-plugin middleware func" line in handleErrorResponse:

  • 408, 418, 429 and 503 log at warn
  • every other 4xx/5xx, including 401 and 403 auth failures, stays at error

This is Option 1 from the issue, which the Visor triage bot endorsed and invited a PR for. It follows the per-error log-level convention already in this file: ErrResponseErrorSent carries errpack.WithLogLevel and gateway/middleware.go honors it via errpack.LogLevel, but the spamming line fires before that wrap and hardcoded .Error(). gateway/mw_jwt.go uses the same switch-then-Log shape (#8401).

No config or API definition schema changes. Deployments alerting on level=error counts from this line will see 408, 418, 429 and 503 move to warn, which is the point of the fix.

Related Issue

Fixes #8245

Motivation and Context

At 1k QPS with 10 percent rejects that is 100 lines/sec of level=error for traffic that is normal by design, which floods severity-based alerting. The JS and gRPC coprocess paths avoid this via ReturnOverrides; the Go-plugin path has no equivalent, and the only control today is the global log_level, which also silences unrelated real errors.

How This Has Been Tested

TestGoPluginMiddleware_handleErrorResponseLogLevel calls handleErrorResponse for 408, 418, 429 and 503 expecting warn, and 400, 401 and 500 expecting error, then checks the returned error still wraps ErrResponseErrorSent with the status code unchanged. 403 is asserted on levelForPluginStatus directly because that branch fires a gateway event.

Locally on Go 1.25: gofmt clean on both files, go build ./gateway/, go vet ./gateway/ and the new test all pass.

Screenshots (if appropriate)

N/A

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Refactoring or add test (improvements in base code or adds test coverage to functionality)

Checklist

  • I ensured that the documentation is up to date
  • I explained why this PR updates go.mod in detail with reasoning why it's required
  • I would like a code coverage CI quality gate exception and have explained why

No documentation or go.mod changes in this PR.

Go-plugin middleware logged every 4xx/5xx plugin response at
level=error, including intentional rejects such as 429 from rate
limiting and 503 from circuit breaking, which floods severity-based
alerting at scale.

Map backpressure status codes (408, 418, 429, 503) to warn for the
"Failed to process request with Go-plugin middleware func" log line
and keep everything else, including auth failures, at error. This
extends the per-error log-level convention already used in this file,
where ErrResponseErrorSent carries errpack.WithLogLevel and
middleware.go logs "Finished" via errpack.LogLevel.

Relates to TykTechnologies#8245
@sentinelone-cnapp-eu1

Copy link
Copy Markdown

SentinelOne CNS Hardcoded Secret Detector
✅ Congratulations, your code is safe

SentinelOne CNS is a cloud-agnostic, agentless CSPM & CWPP solution that continuously detects and prevents vulnerabilities that have the highest probability of being exploited in Azure, AWS, Google Cloud, and Kubernetes.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug] Go-plugin intentional reject (4xx/5xx) emits unconditional level=error log spam — extend existing errpack pattern to fix

1 participant