fix(otlp-exporter-base): drain the fetch response body so that browsers release the keepalive quota - #7002
Conversation
Chromium only releases the request's share of the 64KiB keepalive quota once the response body has been read to its end, and it skips the buffering consumer that would drain the body on its own when the response carries Cache-Control: no-store. Since the transport never read the body, the quota filled up and every later export stayed pending. Read and discard the body before returning, chunk by chunk so that a large response is not buffered. Read failures are logged at debug level and do not change the export outcome. Signed-off-by: Charles Cheng <chengxisheng777@gmail.com>
Pull request dashboard statusWaiting on reviewers · refreshed 2026-08-21 06:16 UTC Review the latest changes. Status above doesn't look right?
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #7002 +/- ##
==========================================
- Coverage 95.02% 95.02% -0.01%
==========================================
Files 409 409
Lines 14298 14317 +19
Branches 3277 3277
==========================================
+ Hits 13587 13605 +18
- Misses 711 712 +1
🚀 New features to boost your workflow:
|
…draining Draining the body left its reader attached, so the stream stayed locked for good. Acquiring a reader on a locked stream throws a `TypeError` without a cause, which is exactly what `isFetchNetworkErrorRetryable` treats as a retryable network error, so any later export handed the same response was retried until it timed out. Release the reader once the body has been read, and acquire it inside the guarded block so a body locked elsewhere cannot decide the export outcome either. Signed-off-by: Charles Cheng <chengxisheng777@gmail.com>
overbalance
left a comment
There was a problem hiding this comment.
Direction and code look good. I want to reproduce #7001 locally against a no-store collector before approving, and could you add a test for the timeout abort firing mid-drain (never-closing body + short timeout) asserting the result still reflects the HTTP status?
The response body is drained while the export timeout is still running, so a collector that holds the body open long enough gets the request aborted in the middle of the read. Add cases for a 200 and a 503 response whose body only ends when the abort fires, asserting the status the collector sent still decides the export outcome.
|
Added in d808444: a They do discriminate the change — with the 94/94 green in the browser suite locally. |
JacksonWeber
left a comment
There was a problem hiding this comment.
Apart from the small lint issue, this LGTM.
|
Thanks for the review! The This branch only touches |
|
Hey, just offering an opinion. This would work as a fix that makes sure the body is read, which is part of the problem for sure. There are some considerations:
We could consider allowing users to set One observation was the library has this protection where it currently reserves 60 out of 64kib of the This satisfies the conditions I think of the issue, and then relying on Chrome to fix the upstream issue if they find it to be a bug. That makes sense too! If you want we could discuss some further options if you guys are interested in some contributions, I'd be happy to help! I've filed an issue with Chromium that they're working through here, https://issues.chromium.org/issues/546438373, it would be really cool to collaborate on this with you! (EDIT: Fixing up my tone, I should have taken a more collaborative approach. If you'll allow me to start over.) |
|
@JacksonWeber CC ^ (Updated above comment) |
|
Thanks for the Chromium report — that's the half of this I couldn't get at from the library side, and it's good to have a tracking bug for it. Your observation that the budget blew before your own byte count reached it lines up with this bug. The transport frees a request's share of the 60KiB it tracks in the A Unrelated, for whoever ends up merging: the |
|
Correction to my last paragraph — the So it isn't a flake to re-run: bringing the branch up to date with |
Which problem is this PR solving?
Fixes #7001
The fetch transport never reads the response body. In Chromium that means the request's share of the 64KiB cumulative
keepalivequota is never given back:HandleLoaderFinish— which decrementsinflight_keepalive_bytes_— is only reachable once the body has been read to its end. Blink normally hides this by draining the body itself through aBufferingBytesConsumer, but it explicitly skips that consumer when the response carriesCache-Control: no-store, which is a common header for a collector to return (fetch_manager.cc).The result, reproduced in the issue, is that successful exports leak the quota until it is exhausted, after which every export stays
(pending)in devtools forever and telemetry is silently lost. The transport's own accounting (pendingBodySize/pendingKeepaliveCount) meanwhile releases at response headers, so it keeps sending requests withkeepalive: truethat the browser can no longer accept.Short description of the changes
response.arrayBuffer(), so a large response is not buffered in memory.finallyblock, the transport now holds its in-flight counters for the lifetime of the body as well. That is intentional: it makes the internal accounting line up with when the browser actually releases the quota, instead of releasing ahead of it.Cancelling the body instead of reading it would not work — in Blink that is the client-abort path, not the read-to-end path that runs
HandleLoaderFinish.The body is still discarded rather than returned as
ExportResponseSuccess.data, so partial-success responses remain unhandled in the browser (otlp-export-delegatehandlesdatawhen the node transport sets it). That is a separate gap and I kept it out of this fix, happy to follow up if you'd like it in the same PR.Type of change
How Has This Been Tested?
npm run test:browserinexperimental/packages/otlp-exporter-base(90 passing). The three new tests fail onmainand pass with the change.npm testin the same package (151 passing), plustest:browserofexporter-trace-otlp-http,exporter-logs-otlp-httpandopentelemetry-exporter-metrics-otlp-http.npm run lint(no new findings).New tests cover that the body is read to its end (and not cancelled) on a successful export, that it is read on a retryable one, and that a body which fails to read leaves the export result untouched.
Checklist: