goplugin: analytics RawResponse is a zero-value HTTP response for plugin-rejected requests (status >= 400)

[tshtark]

Branch / Version

Reproduced against master as of 2026-05-21 (post #7985, which fixed the related but distinct double-write issue).

Summary

When a Go plugin running as a pre/auth_check/post_key_auth/post middleware writes an error response (status >= 400) to the ResponseWriter, the resulting analytics record stored in Redis has RawResponse set to the base64 encoding of a zero-value *http.Response:

HTTP/0.0 000 status code 0
Content-Length: 0

The on-the-wire response is correct (status code, headers, and body that the plugin wrote all reach the client). Only the audit/analytics record is malformed.

AnalyticsRecord.ResponseCode is correct; only RawResponse is the sentinel. As a result, any pump-side processor that parses RawResponse (status line, content-length, headers, body) blows up for every plugin-rejected request, even though those are real, healthy responses the plugin chose to emit.

Reproduce

1. Configure an API with a Go plugin in the pre chain that writes a 4xx response with a body, e.g.:

   func Reject(rw http.ResponseWriter, _ *http.Request) {
       rw.Header().Set("Content-Type", "application/json")
       rw.WriteHeader(http.StatusTooManyRequests)
       _, _ = rw.Write([]byte(`{"reason":"queue_full"}`))
   }

2. Enable analytics_config.enable_detailed_recording = true (so RawResponse is populated at all).

3. Send a request that hits the plugin.

4. Read the resulting analytics record from the Redis LIST analytics-tyk-system-analytics, msgpack-decode it, base64-decode RawResponse.

Observed:

ResponseCode: 429
RawResponse (b64-decoded): "HTTP/0.0 000 status code 0\r\nContent-Length: 0\r\n\r\n"

Expected:

ResponseCode: 429
RawResponse (b64-decoded): "HTTP/1.1 429 Too Many Requests\r\nContent-Type: application/json\r\nContent-Length: 23\r\n\r\n{\"reason\":\"queue_full\"}"

The 2xx admit path on the same plugin emits a correctly serialized HTTP/1.1 200 OK ... RawResponse, so the analytics pipeline itself works — only the goplugin >= 400 branch is affected.

Root cause

Traced in master (post #7985):

1. gateway/mw_go_plugin.go::handlePluginResponse (current line ~307):

   if rw.statusCodeSent >= http.StatusBadRequest {
       return m.handleErrorResponse(r, rw, logger)
   }

   The 2xx branch immediately below calls successHandler.RecordHit(..., rw.getHttpResponse(r), ...) with the wrapped writer's real captured response. The >= 400 branch returns to the base middleware with an error, and the wrapped response is never used for analytics.

2. gateway/mw_go_plugin.go::handleErrorResponse (current line ~322) returns (err, rw.statusCodeSent) with err wrapped in ErrResponseErrorSent when rw.responseSent is true.

3. gateway/middleware.go::createMiddleware (current line ~190) handles that error:

   if err != nil {
       writeResponse := true
       if goPlugin, isGoPlugin := actualMW.(*GoPluginMiddleware); isGoPlugin && goPlugin.handler != nil || errors.Is(err, ErrResponseErrorSent) {
           writeResponse = false
       }
       handler := ErrorHandler{mw.Base()}
       handler.HandleError(w, r, err.Error(), errCode, writeResponse)
       ...
   }

   writeResponse=false correctly prevents HandleError from clobbering the plugin's wire response (this was the fix in [TT-5070] Duplicate WriteHeader calls cause malformed responses from golang plugins #7985).

4. gateway/handler_error.go::HandleError (current line ~99):

   func (e *ErrorHandler) HandleError(w http.ResponseWriter, r *http.Request, errMsg string, errCode int, writeResponse bool) {
       defer e.Base().UpdateRequestSession(r)
       response := &http.Response{}    // zero-value
   
       if writeResponse {
           // populated via writeTemplateErrorResponse / tryWriteOverride / writeJSONRPCErrorResponse
       }
       // ... latency calc, doNotTrack check ...

   With writeResponse=false, response stays the zero-value &http.Response{}.

5. gateway/handler_error.go (current line ~232):

   var wireFormatRes bytes.Buffer
   response.Write(&wireFormatRes)
   rawResponse = base64.StdEncoding.EncodeToString(wireFormatRes.Bytes())

   (&http.Response{}).Write(buf) serializes the zero-value into exactly HTTP/0.0 000 status code 0\r\nContent-Length: 0\r\n\r\n — that is the bytes that get base64'd into record.RawResponse.

The 2xx branch dodges this by recording analytics via successHandler.RecordHit(..., rw.getHttpResponse(r), ...) inside handlePluginResponse and never routing through HandleError. The >= 400 branch doesn't have an equivalent.

Why obvious workarounds don't apply

Option	Verdict
analytics_config.enable_detailed_recording toggle	Already enabled. The 200 path proves capture works on the same code path; the 4xx path simply bypasses capture.
Move plugin to a different goplugin hook position (auth_check, post_key_auth, post)	All goplugin hooks route through handlePluginResponse → same >= 400 branch → same bug.
Move to response hook	response only fires after upstream proxy. Plugins that reject before proxy (rate limiting, admission control, auth) can't move there.
Tyk analytics plugin (post-record processor)	Has access to the AnalyticsRecord but not the original ResponseWriter; cannot reconstruct a faithful RawResponse, only synthesize a plausible one.
Emit a custom analytics record from the plugin	Couples plugin source to Tyk's analytics record shape; defeats the point of the analytics layer.

Proposed fix

Stash the wrapped writer's *http.Response in the request context inside handlePluginResponse, and have HandleError use it when writeResponse=false instead of the zero-value placeholder.

The wrapped writer already constructs a real *http.Response for the 2xx analytics path (customResponseWriter.getHttpResponse(r) at mw_go_plugin.go line ~83). The fix is to reuse that for the 4xx path.

Diff sketch (4 files):

// ctx/ctx.go — add to the Key iota
const (
    // ... existing keys ...

    // CapturedResponse holds a *http.Response that was constructed by an
    // upstream middleware (currently goplugin) directly from a wrapped
    // ResponseWriter. ErrorHandler.HandleError reads this to populate the
    // analytics RawResponse when writeResponse=false, instead of serializing
    // a zero-value http.Response (which renders as "HTTP/0.0 000 status code 0").
    CapturedResponse
)

// gateway/api.go — alongside the other ctxSet/ctxGet helpers
func ctxSetCapturedResponse(r *http.Request, resp *http.Response) {
    setCtxValue(r, ctx.CapturedResponse, resp)
}

func ctxGetCapturedResponse(r *http.Request) *http.Response {
    if v, ok := r.Context().Value(ctx.CapturedResponse).(*http.Response); ok {
        return v
    }
    return nil
}

// gateway/mw_go_plugin.go::handlePluginResponse — current line ~307
if rw.statusCodeSent >= http.StatusBadRequest {
    // The base middleware will route this through ErrorHandler.HandleError
    // with writeResponse=false (to avoid clobbering the wire response the
    // plugin already wrote). Without help, HandleError will serialize a
    // zero-value *http.Response into RawResponse — see
    // gateway/handler_error.go line ~232 — producing
    //   "HTTP/0.0 000 status code 0\r\nContent-Length: 0\r\n\r\n"
    // in every plugin-error analytics record. Stash the real response here
    // so HandleError can use it.
    ctxSetCapturedResponse(r, rw.getHttpResponse(r))
    return m.handleErrorResponse(r, rw, logger)
}

// gateway/handler_error.go::HandleError — replace the response init block
func (e *ErrorHandler) HandleError(w http.ResponseWriter, r *http.Request, errMsg string, errCode int, writeResponse bool) {
    defer e.Base().UpdateRequestSession(r)
    response := &http.Response{}

    if writeResponse {
        if e.Spec.IsMCP() && e.shouldWriteJSONRPCError(r) {
            response = e.writeJSONRPCErrorResponse(w, r, errMsg, errCode)
        } else if resp := e.tryWriteOverride(w, r, errMsg, errCode); resp != nil {
            response = resp
        } else {
            response = e.writeTemplateErrorResponse(w, r, errMsg, errCode)
        }
    } else if captured := ctxGetCapturedResponse(r); captured != nil {
        // A middleware (e.g. goplugin) already wrote the wire response and
        // stashed its captured *http.Response here. Use it so the analytics
        // RawResponse reflects the real bytes the client received.
        response = captured
    }

    // ... rest unchanged ...

Why this approach

• Strictly additive. Any middleware path that doesn't stash a response (everything except goplugin >= 400 today) sees the existing zero-value behavior — no regression.
• No new sentinel error types, no changes to HandleError's signature or return type, no semantic overload of DoNotTrack.
• Mirrors the 2xx path: both branches now use rw.getHttpResponse(r) as the source of truth for the analytics record's response shape.
• The responseCode fallback at handler_error.go line ~183 already handles response.StatusCode != 0, so it correctly picks up the real status code from the stashed response (replacing the previous fallback to errCode).

Suggested test

Mirroring the test added in #7985 in goplugin/mw_go_plugin_test.go. Asserts the analytics record's RawResponse decodes to a parseable HTTP/1.1 status line carrying the plugin's status code (anti-witness: rejects HTTP/0.0 000).

func TestGoPlugin_AnalyticsRawResponseCapturesPluginResponse(t *testing.T) {
    ts := gateway.StartTest(nil)
    t.Cleanup(ts.Close)

    ts.Gw.BuildAndLoadAPI(func(spec *gateway.APISpec) {
        spec.Proxy.ListenPath = "/test-api/"
        spec.UseKeylessAccess = true
        spec.UseStandardAuth = false
        spec.EnableDetailedRecording = true
        spec.CustomMiddleware = apidef.MiddlewareSection{
            Driver: apidef.GoPluginDriver,
            Pre: []apidef.MiddlewareDefinition{{
                Name: "RejectWithBody",
                Path: goPluginFilename(),
            }},
        }
    })

    redisAnalyticsKeyName := analyticsKeyName + ts.Gw.Analytics.analyticsSerializer.GetSuffix()
    ts.Gw.Analytics.Store.GetAndDeleteSet(redisAnalyticsKeyName)

    ts.Run(t, test.TestCase{
        Path: "/test-api/get",
        Code: http.StatusForbidden,
    })

    ts.Gw.Analytics.Flush()
    results := ts.Gw.Analytics.Store.GetAndDeleteSet(redisAnalyticsKeyName)
    assert.Len(t, results, 1)

    var record analytics.AnalyticsRecord
    err := ts.Gw.Analytics.analyticsSerializer.Decode([]byte(results[0].(string)), &record)
    assert.NoError(t, err)
    assert.Equal(t, http.StatusForbidden, record.ResponseCode)

    raw, err := base64.StdEncoding.DecodeString(record.RawResponse)
    assert.NoError(t, err)
    assert.True(t, strings.HasPrefix(string(raw), "HTTP/1.1 403 "),
        "RawResponse should start with HTTP/1.1 403, got: %q", truncate(string(raw), 80))
    assert.False(t, strings.HasPrefix(string(raw), "HTTP/0.0 000"),
        "RawResponse must not be the zero-value sentinel: %q", truncate(string(raw), 80))
    assert.Contains(t, string(raw), "hello",
        "RawResponse should contain the body the plugin wrote")
}

RejectWithBody plugin already exists in test/goplugins/test_goplugin.go as of #7985.

Impact

• Any downstream that parses AnalyticsRecord.RawResponse (response-header dashboards, body inspection, response-size analytics) is wrong for every plugin-rejected request. The wire response itself is unaffected — only the audit trail is malformed.
• Affects all goplugin-based admission/auth/rate-limit/policy plugins that emit a body on reject.
• We currently work around this in our own e2e test by softening the RawResponse assertion to a t.Logf inversion-witness; we'd like to remove that softener once this is fixed upstream.

Related

• [TT-5070] Duplicate WriteHeader calls cause malformed responses from golang plugins #7985 (merged 2026-04-22) fixed the related but distinct issue of HandleError clobbering the wire body. That PR did not touch the analytics path; RawResponse is still serialized from the zero-value *http.Response after that fix.

Happy to send the patch as a PR once a maintainer signals interest in the approach.

[probelabs]

Hello @tshtark,

Thank you for this exceptionally detailed and well-researched bug report. The clarity of the summary, reproduction steps, and root cause analysis is incredibly helpful.

Understanding the Issue

Based on your report, I understand the following:

• When a Go plugin middleware (e.g., in the pre hook) rejects a request by writing a response with a status code >= 400, the actual response sent to the client is correct.
• However, the corresponding analytics record stored by Tyk contains a malformed RawResponse. Specifically, it's the base64 encoding of a zero-value *http.Response, not the actual response bytes.
• This issue specifically affects the analytics data pipeline for any consumer that needs to parse the full raw response of rejected requests.

Analysis

Your root cause analysis appears to be spot on. The control flow for plugin-generated error responses correctly prevents the gateway from sending a second response body (the fix from #7985), but as a side effect, the ErrorHandler that generates the analytics data is left with a zero-value *http.Response object to serialize.

The proposed solution of using the request context to pass the captured *http.Response from the Go plugin middleware to the ErrorHandler is an excellent and clean approach. It directly addresses the problem by providing the missing data to the analytics recording step without introducing breaking changes or complex logic.

Next Steps

The proposed fix seems very promising. We encourage you to open a Pull Request with the changes and the test case you've outlined. The team will then review it as part of the standard contribution process.

Thank you again for your thorough investigation and contribution!

References

• gateway/mw_go_plugin.go
• gateway/handler_error.go
• gateway/middleware.go
• ctx/ctx.go

---

Powered by Visor from Probelabs

💡 TIP: You can chat with Visor using /visor ask <your question>