Streaming debug_traceBlock with inline tracer to prevent OOM#9844
Closed
qu0b wants to merge 4 commits intobesu-eth:bal-devnet-2-with-prefetchfrom
Closed
Streaming debug_traceBlock with inline tracer to prevent OOM#9844qu0b wants to merge 4 commits intobesu-eth:bal-devnet-2-with-prefetchfrom
qu0b wants to merge 4 commits intobesu-eth:bal-devnet-2-with-prefetchfrom
Conversation
debug_traceBlockByHash/ByNumber/Block accumulated all transaction traces in a synchronized ArrayList before serializing the entire response. A single block trace produces 350-600 MB of JSON (with disableMemory=false), causing OOM under concurrent requests on nodes targeted by tracing services like tracoor. Introduce StreamingJsonRpcSuccessResponse which writes each transaction trace directly to the HTTP response via JsonGenerator as it is produced. Peak memory drops from O(all_traces_in_block) to O(single_trace). Tested on bal-devnet-2: 26+ blocks traced continuously under 12 GiB memory limit, 12+ GB total data streamed, memory stable at 2-4 GiB. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Instead of accumulating all TraceFrames in memory per transaction and then converting them to StructLog objects, write each StructLog directly to the JsonGenerator as it is produced during execution. After each transaction's frames are written, reset the tracer to release memory. This reduces peak memory from O(all_frames_in_block) to O(one_tx_frames), preventing OOM when tracoor sends concurrent debug_traceBlockByHash requests with memory capture enabled. Stress tested with 10 concurrent block traces: peak 4.1 GiB vs 7.6 GiB before, with full recovery after GC. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
qu0b
changed the title
qu0b
added a commit
to ethpandaops/bal-devnets
that referenced
this pull request
Feb 19, 2026
Limit `--rpc-http-max-active-connections` to 20 (default 80) and increase `--Xhttp-timeout-seconds` to 600 (default 30) on all besu nodes. tracoor floods 250+ concurrent `debug_traceBlockByHash` requests with full memory/storage tracing enabled. Even with the streaming fix (PR besu-eth/besu#9844), the default 80-connection limit allows enough concurrent traces to exhaust JVM heap. Lowering to 20 prevents OOM while still serving normal RPC traffic. The 600s timeout prevents heavy block traces from being killed mid-flight. Already deployed manually to both live besu nodes — stable with zero OOM under continuous tracoor load. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace frame-by-frame iteration with true inline streaming: StructLog entries are written directly to the JsonGenerator during EVM execution via StreamingDebugOperationTracer, achieving O(1) frame memory instead of O(all_frames_per_tx). Additionally implements lazy memory capture: EVM memory is only copied for the 24 opcodes that actually read/write memory (MLOAD, MSTORE, CALL, CREATE, LOG*, etc). All other opcodes (~90%) omit the memory field entirely, reducing per-frame overhead from ~256KB to ~1-2KB. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
When debug_traceBlockByHash fails mid-stream (connection reset or worldstate unavailable), the JSON generator may be in an unknown nested context. Previously this caused JsonGenerationException and Self-suppression errors. - Move writeStartArray before processTracing so worldstate-not-found produces valid empty array [] instead of crashing - Track whether the tracing lambda ran to detect mid-stream failures - Swallow generator.close() IOException in handleJsonObjectResponse to prevent Self-suppression not permitted errors Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
qu0b
changed the title
usmansaleem
reviewed
Feb 24, 2026
| .thenProcess("executeTransaction", executeTransactionStep) | ||
| .thenProcessAsyncOrdered( | ||
| "debugTraceTransactionStep", | ||
| DebugTraceTransactionStepFactory.createAsync( |
Contributor
There was a problem hiding this comment.
we need a plan to migrate DebugTraceTransactionStepFactory.createAsync( traceOptions, protocolSpec) because this is where other tracer types (callTracer etc.) got converted. Ideally, other tracer types should also directly implement OperationTracer.
ahamlat
reviewed
Feb 26, 2026
|
|
||
| // memory — only capture for memory-touching opcodes | ||
| if (options.traceMemory() && frame.memoryWordSize() > 0) { | ||
| if (MEMORY_OPCODES.contains(opcodeNumber)) { |
Contributor
There was a problem hiding this comment.
I believe this is not correct. Even if the opcode didn't touch memory, if there is data in memory, it should be displayed.
Author
There was a problem hiding this comment.
Not sure what you mean? This does not prevent displaying anything?
Author
There was a problem hiding this comment.
Data can only get into memory through memory changing opcodes, so we don't need to capture memory if it hasn't changed, we can reference the previous opcode memory state.
Contributor
|
closing in favour of #9848 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
debug_traceBlockByHashaccumulates allTraceFrameobjects in anArrayListfor every transaction in a block before serializing. Each frame holds a full copy of EVM memory (up to 256KB). On blocks with 200K-600K+ opcode steps, this exhausts heap and causesOutOfMemoryError— particularly under continuous load from tracing services like tracoor.Solution
New
StreamingDebugOperationTracerwrites each struct log entry directly to aJsonGeneratorduring EVM execution viatracePostExecution(), then discards it immediately. No frames are accumulated. Combined with lazy memory capture (only 22 of ~140 opcodes actually touch memory), peak memory drops from O(all_frames_in_block) to O(1).The pipeline architecture (
PipelineBuilder/ExecuteTransactionStep/DebugTraceTransactionStepFactory) is removed in favor of direct sequential iteration with inline JSON serialization.Error handling
JsonRpcObjectExecutorhandlesgenerator.close()failures on connection reset to preventSelf-suppression not permittedexceptionsChanges
StreamingDebugOperationTracer.javaStreamingJsonRpcSuccessResponse.javaJsonRpcResponsethat streamsresultviaJsonGeneratorAbstractDebugTraceBlock.javagetStreamingTraces()with mid-stream error handlingDebugTraceBlockByNumber.javaDebugOperationTracer(per-tx reset)JsonRpcObjectExecutor.javaStructLogWithError.javaVerification
pc,op,gas,gasCost,depth,memory,storage)🤖 Generated with Claude Code