Stream debug_traceBlock* responses directly to avoid OOM on large blocks

[daniellehrner]

PR description

Converts debug_traceBlockByNumber, debug_traceBlockByHash, and `debug_traceBlock from accumulate-then-serialize to stream-as-you-go. Previously, these methods built the entire JSON response in memory (via TransactionTrace + DebugTraceTransactionResult), which OOMs on blocks with many transactions or complex traces. Now, structLogs are written directly to the HTTP/WebSocket output stream during EVM execution.

Infrastructure changes:

• New StreamingJsonRpcMethod interface: streaming methods implement streamResponse(request, outputStream, mapper) instead of response(request)
• JsonRpcMethod.isStreaming() marker allows the HTTP/WS handlers to dispatch to the streaming path
• streamProcess() added to the JsonRpcProcessor chain (Base, Timed, Traced, Authenticated) so streaming requests get the same metrics, tracing, and auth as regular requests
• JsonRpcObjectExecutor and WebSocketMessageHandler detect streaming methods and write directly to the response stream
• JsonRpcExecutor shared preamble extracted into prepareExecution() to eliminate duplication between execute() and executeStreaming()

Trace-specific changes

• AbstractDebugTraceBlock base class with shared option parsing and response writing
• DebugTraceBlockStreamer — the core streaming engine. For OPCODE_TRACER, writes structLogs during execution via a frame callback. For other tracers (callTracer, etc.), accumulates per-transaction then serializes
• DebugOperationTracer.StreamingFrameWriter — zero-allocation callback that passes raw trace data + live MessageFrame reference, bypassing TraceFrame/Builder/Optional construction entirely
• Reusable StringBuilder for hex conversion (StructLog.toCompactHex overload), memory read via readMutableMemory() (view, no copy), storage read directly from getUpdatedStorage() — eliminates ~130+ transient objects per
  opcode step
• EthScheduler dependency removed from debug trace method constructors (no longer needed without async dispatch)

Breaking changes

JSON field ordering changed for debug_traceBlock* with OPCODE_TRACER (default tracer):

Before:

  {"txHash": "0x...", "result": {"gas": 21000, "failed": false, "returnValue": "", "structLogs": [...]}}

After:

  {"txHash": "0x...", "result": {"structLogs": [...], "gas": 21000, "failed": false, "returnValue": ""}}

gas, failed, and returnValue now appear after structLogs because they're only known after execution completes. JSON-RPC clients that parse by key name (standard) are unaffected. Clients that depend on field ordering will
break.

Batch requests containing streaming methods now return {"error": {"code": -32600, "message": "Invalid request"}} for those methods instead of crashing the batch. This is new behavior but I am unsure if tracing more than one block in parallel would have been possible without OOM anyways. Batch requests fall back to the normal OperationTracer which accumulates the results in memory

Performance tests

I ran the following script to trace 10 recent blocks in a row to compare the current implementation against this PR. The script was:

#!/bin/bash
RPC_URL="http://localhost:8545"

if [ -z "$1" ]; then
  echo "Usage: $0 <start_block>" >&2
  exit 1
fi

START="$1"

echo "Testing debug_traceBlockByNumber streaming (10 blocks starting at $START)"
echo "RPC: $RPC_URL"
echo "---"
printf "%-12s %10s %10s %12s\n" "Block" "Total(s)" "TTFB(s)" "Size(MB)"
echo "---"

total_time=0
total_size=0

for i in $(seq 0 9); do
  block=$(printf "0x%X" $(( START + i )))
  result=$(curl -o /dev/null -w "%{time_total} %{time_starttransfer} %{size_download}" \
    -s -X POST "$RPC_URL" \
    -H 'Content-Type: application/json' \
    -d "{\"jsonrpc\":\"2.0\",\"method\":\"debug_traceBlockByNumber\",\"params\":[\"$block\"],\"id\":1}")
  time=$(echo "$result" | awk '{print $1}')
  ttfb=$(echo "$result" | awk '{print $2}')
  size=$(echo "$result" | awk '{print $3}')
  mb=$(echo "scale=2; $size/1048576" | bc)
  printf "%-12s %10.3f %10.3f %10.2fMB\n" "$block" "$time" "$ttfb" "$mb"
  total_time=$(echo "$total_time + $time" | bc)
  total_size=$(echo "$total_size + $size" | bc)
done

avg_time=$(echo "scale=3; $total_time/10" | bc)
avg_mb=$(echo "scale=2; $total_size/10/1048576" | bc)
echo "---"
printf "%-12s %10.3f %10s %10.2fMB\n" "Average" "$avg_time" "" "$avg_mb"
printf "%-12s %10.3f %10s %10.2fMB\n" "Total" "$total_time" "" "$(echo "scale=2; $total_size/1048576" | bc)"

On the feature node, which includes this PR we got:

./test_trace_block.sh 24496058
Testing debug_traceBlockByNumber streaming (10 blocks starting at 24496058)
RPC: http://localhost:8545
---
Block          Total(s)    TTFB(s)     Size(MB)
---
0x175C7BA         6.602      0.020     942.23MB
0x175C7BB        12.770      0.004    1860.28MB
0x175C7BC         4.399      0.003     524.80MB
0x175C7BD         5.358      0.003     643.87MB
0x175C7BE        13.290      0.003    2283.97MB
0x175C7BF        15.382      0.003    2586.28MB
0x175C7C0         2.897      0.002     349.54MB
0x175C7C1         4.722      0.002     615.65MB
0x175C7C2         4.555      0.002     604.91MB
0x175C7C3        16.937      0.004    2753.11MB
---
Average           8.691               1316.46MB
Total            86.913              13164.68MB

On the control node, which is main we got:

./test_trace_block.sh 24496058
Testing debug_traceBlockByNumber streaming (10 blocks starting at 24496058)
RPC: http://localhost:8545
---
Block          Total(s)    TTFB(s)     Size(MB)
---
0x175C7BA         9.488      6.427     942.23MB
0x175C7BB        11.434      4.777    1860.28MB
0x175C7BC         4.623      2.952     524.80MB
0x175C7BD         5.753      3.696     643.87MB
0x175C7BE        13.923      5.766    2283.97MB
0x175C7BF        30.170     30.170       0.00MB
0x175C7C0        39.813     39.813       0.00MB
0x175C7C1        38.678     38.678       0.00MB
^C

The control node crashed during the execution, as can be seen by retruning 0 bytes on the bottom 3 blocks.

TTFB (time to first byte) is only a few ms on the feature node and several seconds on the control node, showing that the streaming works correctly and starts to send data almost immediately.

The total response time varies between the two without a clear winner.

During the tests we saw the following memory consumption:

We see the expected spikes on the control node in GC time and general memory consumption. As the memory consumption increases by several GBs we eventually run into a OOM error which crahes the node.

On the feature node GC time increases a bit, but the general memory consumption stays relatively flat, as expected because the streaming only keeps very little data in memory before writing it to the socket and deleting it right away.

Fixed Issue(s)

Thanks for sending a pull request! Have you done the following?

• Checked out our contribution guidelines?
• Considered documentation and added the doc-change-required label to this PR if updates are required.
• Considered the changelog and included an update if required.
• For database changes (e.g. KeyValueSegmentIdentifier) considered compatibility and performed forwards and backwards compatibility tests

Locally, you can run these tests to catch failures early:

• spotless: ./gradlew spotlessApply
• unit tests: ./gradlew build
• acceptance tests: ./gradlew acceptanceTest
• integration tests: ./gradlew integrationTest
• reference tests: ./gradlew ethereum:referenceTests:referenceTests
• hive tests: Engine or other RPCs modified?

[ahamlat]

@daniellehrner is this PR ready for review ?

[ahamlat]

interesting 👍 , not very important/urgent, but it would be great to see JMH benchmarks on both methods.

[ahamlat]

I think it makes sens to have two different DebugOperationTracer, one that uses streaming and one that doesn't use streaming. It is a bit misleading to have traceFrames object where it is not needed anymore for the streaming version.
Also, fix unit tests as some of them are failing related to this PR.

[Copilot]

Pull request overview

This PR adds first-class support for streaming JSON-RPC responses, then migrates debug_traceBlock* (especially opcode tracing) to stream structLogs as they are produced to avoid OOM on large blocks.

Changes:

• Introduces a streaming execution path (StreamingJsonRpcMethod, executor/processor support) for HTTP and WebSocket handlers.
• Adds streaming-capable debug tracers and a DebugTraceBlockStreamer that writes opcode struct logs during EVM execution.
• Updates debug trace block golden files and rewrites debug trace tests to validate the streamed JSON output.

Reviewed changes

Copilot reviewed 30 out of 30 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
ethereum/core/src/main/java/org/hyperledger/besu/ethereum/vm/StreamingDebugOperationTracer.java	New operation tracer that emits per-opcode data via a callback instead of accumulating frames.
ethereum/core/src/main/java/org/hyperledger/besu/ethereum/vm/DebugOperationTracer.java	Refactors to share logic via AbstractDebugOperationTracer and optionally stream frames via a consumer.
ethereum/core/src/main/java/org/hyperledger/besu/ethereum/vm/AbstractDebugOperationTracer.java	New base class for shared opcode filtering, stack capture, and gas-cost computation.
ethereum/api/src/test/resources/org/hyperledger/besu/ethereum/api/jsonrpc/trace/chain-data/genesis-osaka.json	Adds a genesis config for updated trace tests.
ethereum/api/src/test/resources/org/hyperledger/besu/ethereum/api/jsonrpc/debug/trace-block/debug_traceBlock_disableStorage.json	Updates expected JSON ordering for streamed results.
ethereum/api/src/test/resources/org/hyperledger/besu/ethereum/api/jsonrpc/debug/trace-block/debug_traceBlock_disableStack.json	Updates expected JSON ordering for streamed results.
ethereum/api/src/test/resources/org/hyperledger/besu/ethereum/api/jsonrpc/debug/trace-block/debug_traceBlock_disableMemory.json	Updates expected JSON ordering for streamed results.
ethereum/api/src/test/resources/org/hyperledger/besu/ethereum/api/jsonrpc/debug/trace-block/debug_traceBlock_default.json	Updates expected JSON ordering for streamed results.
ethereum/api/src/test/java/org/hyperledger/besu/ethereum/api/jsonrpc/internal/methods/DebugTraceBlockTest.java	Rewrites tests to invoke streamResponse and validate streamed JSON content.
ethereum/api/src/test/java/org/hyperledger/besu/ethereum/api/jsonrpc/internal/methods/DebugTraceBlockByNumberTest.java	Rewrites tests to invoke streamResponse and validate streamed JSON content.
ethereum/api/src/test/java/org/hyperledger/besu/ethereum/api/jsonrpc/internal/methods/DebugTraceBlockByHashTest.java	Rewrites tests to invoke streamResponse and validate streamed JSON content.
ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/jsonrpc/websocket/WebSocketMessageHandler.java	Detects streaming methods and writes directly to the WebSocket output stream.
ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/jsonrpc/methods/JsonRpcMethodsFactory.java	Updates debug method construction after removing EthScheduler dependency.
ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/jsonrpc/methods/DebugJsonRpcMethods.java	Removes EthScheduler usage and wires new streaming debug trace methods.
ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/jsonrpc/internal/results/StructLog.java	Adds toCompactHex(..., StringBuilder) overload to reduce allocations.
ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/jsonrpc/internal/methods/StreamingJsonRpcMethod.java	New interface for methods that stream directly to an OutputStream.
ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/jsonrpc/internal/methods/JsonRpcMethod.java	Adds an isStreaming() marker method.
ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/jsonrpc/internal/methods/DebugTraceBlockStreamer.java	New streaming engine that writes opcode structLogs while executing transactions.
ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/jsonrpc/internal/methods/DebugTraceBlockByNumber.java	Implements streaming for debug_traceBlockByNumber with a fallback batch path.
ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/jsonrpc/internal/methods/DebugTraceBlockByHash.java	Implements streaming for debug_traceBlockByHash.
ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/jsonrpc/internal/methods/DebugTraceBlock.java	Implements streaming for debug_traceBlock and adjusts invalid/parent-missing handling.
ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/jsonrpc/internal/methods/AbstractDebugTraceBlock.java	Shared streaming/batch logic for debug trace block methods and response writing.
ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/jsonrpc/execution/TracedJsonRpcProcessor.java	Adds streamProcess() support while preserving metrics/tracing behavior.
ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/jsonrpc/execution/TimedJsonRpcProcessor.java	Adds timing support for streamed execution.
ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/jsonrpc/execution/JsonRpcProcessor.java	Adds streamProcess() hook to processor chain.
ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/jsonrpc/execution/JsonRpcExecutor.java	Adds streaming dispatch and refactors shared execution preparation.
ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/jsonrpc/execution/BaseJsonRpcProcessor.java	Implements streamed execution including error mapping for invalid params/runtime errors.
ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/jsonrpc/execution/AuthenticatedJsonRpcProcessor.java	Adds auth checks for streamed execution.
ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/handlers/JsonRpcObjectExecutor.java	Routes streaming methods to executeStreaming() and writes directly to response stream.
ethereum/api/src/main/java/org/hyperledger/besu/ethereum/api/handlers/AbstractJsonRpcExecutor.java	Makes SPAN_CONTEXT available to subclasses for streaming execution.

---

You can also share your feedback on Copilot code review. Take the survey.

[macfarla]

writeStreamingStructLog creates new TreeMap<>(updatedStorage) on every traced opcode. The class already has private final TreeMap<UInt256, UInt256> sortedStorage — that field should be cleared and re-used (sortedStorage.clear(); sortedStorage.putAll(updatedStorage)) instead of allocating a new one each time. For a block with thousands of transactions and complex storage access this dominates GC pressure.

[ahamlat]

My understanding is this happens only when there're errors : JsonRpcErrorResponse with Invalid request. When reading the code, it seems weird for me that preparing a request returned a response instead of the PreparedRequest.
I would make prepareExecution return only PreparedRequest and throw IllegalArgumentException to be handled by executeStreaming, and return optional.of(JsonRpcErrorResponse(id, INVALID_REQUEST))

[ahamlat]

Testing bigger buffer size 256 KiB with 1 MiB max queue size showed slightly better latency numbers but was less stable then this configuration. I suggest to keep existing configuration re-evaluate later if needed.

[ahamlat]

LGTM, I tested different blocks on mainnet and noticed big improvements in terms of performance. The most interesting ones are when existing besu nodes (before this PR) couldn't reply because of OOO errors where this PR is able to reply with very small memory overhead.
There are small comments but nothing blocking. We can re-evaluate in future PRs.