Skip to content

Commit 364b9ad

Browse files
vincent-k2026vincent-k2026
authored
docs(node): add debug-trace-server guide (#47)
## Summary Adds `docs/node/debug-trace-server.md` — operator guide for self-hosting a standalone `debug_*` / `trace_*` RPC server backed by SALT stateless execution. - **When to use this**: explains when to self-host vs. using the public RPC trace methods. - **Quick start**: Mainnet/Testnet commands with parameter explanations; clarifies that `--witness-endpoint` can point at the public RPC for teams without their own witness generator. - **Production setup**: local cache mode with `--data-dir`, block pruning, response cache, metrics. - **Supported RPC methods**: Geth-style (`debug_traceBlockByNumber`, `debug_traceBlockByHash`, `debug_traceTransaction`) and Parity-style (`trace_block`, `trace_transaction`) with full tracer table. - **Command-line reference**: all flags with env vars, defaults, and descriptions — grouped by core, local cache, timeout/concurrency, response cache, monitoring. - **Deployment examples**: systemd unit and env-var file for containers. - **Troubleshooting**: database reset, upstream failures, memory, latency. Updates `docs/SUMMARY.md` to list the new page under Node Operation.
1 parent b0b06ab commit 364b9ad

3 files changed

Lines changed: 291 additions & 1 deletion

File tree

‎docs/SUMMARY.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,3 +46,4 @@
4646
- [Stateless Validation](node/stateless-validation.md)
4747
- [Validator Architecture](node/validator-architecture.md)
4848
- [Get Block Witness](node/witness.md)
49+
- [Debug Trace Server](node/debug-trace-server.md)

‎docs/node/debug-trace-server.md‎

Lines changed: 289 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,289 @@
1+
---
2+
description: Self-host debug_* and trace_* RPC methods for MegaETH by replaying blocks with SALT witnesses — no archive node required.
3+
---
4+
5+
# Debug trace server
6+
7+
The **debug-trace-server** provides `debug_*` and `trace_*` JSON-RPC methods for MegaETH.
8+
It re-executes blocks against [SALT](https://github.com/megaeth-labs/salt) witness data instead of a full state database, so it needs no archive node and runs on commodity hardware.
9+
10+
Point it at any MegaETH RPC endpoint, and it can trace any block or transaction that endpoint serves.
11+
12+
## When to use this
13+
14+
Run your own debug-trace-server when you need a **self-hosted** trace endpoint — for example, to power a block explorer backend, an indexer, or high-volume trace queries that would exceed public RPC rate limits.
15+
16+
If you only need occasional traces, the public MegaETH RPC already serves `debug_traceTransaction` and related methods directly — see [Debugging Transactions](../dev/send-tx/debugging.md).
17+
18+
## Installation
19+
20+
The server is built from source alongside the [stateless validator](stateless-validation.md).
21+
22+
```bash
23+
git clone https://github.com/megaeth-labs/stateless-validator.git
24+
cd stateless-validator
25+
cargo build --release --bin debug-trace-server
26+
```
27+
28+
The binary is at `./target/release/debug-trace-server`.
29+
The project pins a nightly Rust toolchain via `rust-toolchain.toml`; `cargo build` downloads it automatically on first run.
30+
31+
## Quick start
32+
33+
The server needs two upstream endpoints to operate:
34+
35+
| Flag | What it connects to | Methods called |
36+
| -------------------- | ----------------------------------------------------------------- | ------------------------------------------------------------------ |
37+
| `--rpc-endpoint` | A MegaETH JSON-RPC node that serves standard Ethereum block data. | `eth_getBlockByNumber`, `eth_getHeaderByHash`, `eth_getCodeByHash` |
38+
| `--witness-endpoint` | A MegaETH endpoint that serves SALT block witnesses. | [`mega_getBlockWitness`](witness.md) |
39+
40+
{% hint style="info" %}
41+
Most external teams do not run their own witness generator.
42+
Point `--witness-endpoint` at the MegaETH public RPC — it serves `mega_getBlockWitness` for all historical blocks.
43+
In the common case both flags point to the same URL.
44+
{% endhint %}
45+
46+
{% tabs %}
47+
{% tab title="Mainnet" %}
48+
49+
```bash
50+
./target/release/debug-trace-server \
51+
--rpc-endpoint https://mainnet.megaeth.com/rpc \
52+
--witness-endpoint https://mainnet.megaeth.com/rpc
53+
```
54+
55+
{% endtab %}
56+
57+
{% tab title="Testnet" %}
58+
59+
```bash
60+
./target/release/debug-trace-server \
61+
--rpc-endpoint https://carrot.megaeth.com/rpc \
62+
--witness-endpoint https://carrot.megaeth.com/rpc
63+
```
64+
65+
{% endtab %}
66+
{% endtabs %}
67+
68+
The server listens on `0.0.0.0:8545` by default.
69+
Verify it is running:
70+
71+
```bash
72+
curl -s http://localhost:8545 \
73+
-H 'Content-Type: application/json' \
74+
-d '{"jsonrpc":"2.0","method":"debug_traceBlockByNumber","params":["latest",{"tracer":"callTracer"}],"id":1}' \
75+
| head -c 200
76+
```
77+
78+
### Production setup
79+
80+
For production traffic, add `--data-dir` to enable **local cache mode**.
81+
The server starts a background pipeline that pre-fetches blocks and witnesses into a local database, so trace requests read locally instead of hitting upstream RPC on every call.
82+
83+
```bash
84+
./target/release/debug-trace-server \
85+
--rpc-endpoint https://mainnet.megaeth.com/rpc \
86+
--witness-endpoint https://mainnet.megaeth.com/rpc \
87+
--data-dir ./dts-data \
88+
--blocks-to-keep 5000 \
89+
--response-cache-max-size 2GB \
90+
--metrics-enabled
91+
```
92+
93+
| Flag | Effect |
94+
| ------------------------------- | ------------------------------------------------------------------------------------------------- |
95+
| `--data-dir ./dts-data` | Enable local cache mode. Creates a `trace_server.redb` database and starts background block sync. |
96+
| `--blocks-to-keep 5000` | Retain the most recent 5 000 blocks; older blocks are pruned automatically. |
97+
| `--response-cache-max-size 2GB` | Cache serialized trace responses in memory so repeated requests return instantly. |
98+
| `--metrics-enabled` | Expose a Prometheus `/metrics` endpoint on port 9090. |
99+
100+
{% hint style="warning" %}
101+
On the first start with `--data-dir`, the server fetches the latest block from upstream as an anchor and begins syncing forward.
102+
To anchor at a specific block, pass `--start-block <BLOCK_HASH>` — this accepts a block **hash**, not a number.
103+
{% endhint %}
104+
105+
## Supported RPC methods
106+
107+
### Geth-style (`debug_*`)
108+
109+
| Method | Description |
110+
| -------------------------- | --------------------------------------------- |
111+
| `debug_traceBlockByNumber` | Trace all transactions in a block, by number. |
112+
| `debug_traceBlockByHash` | Trace all transactions in a block, by hash. |
113+
| `debug_traceTransaction` | Trace a single transaction by hash. |
114+
| `debug_getCacheStatus` | Query current response cache statistics. |
115+
116+
The trace methods accept an optional second parameter with [Geth debug tracing options](https://geth.ethereum.org/docs/developers/evm-tracing/built-in-tracers):
117+
118+
```json
119+
{
120+
"method": "debug_traceBlockByNumber",
121+
"params": ["latest", { "tracer": "callTracer" }]
122+
}
123+
```
124+
125+
Supported tracers:
126+
127+
| Tracer | `tracer` value | Output |
128+
| ----------------------- | ------------------ | ----------------------------------------------------------------------------- |
129+
| Default (struct logger) | _(omit)_ | Opcode-level trace with gas, stack, memory, storage at each step. |
130+
| Call tracer | `"callTracer"` | Nested call tree with inputs, outputs, gas per frame. |
131+
| Prestate tracer | `"prestateTracer"` | Account state before execution; add `"diffMode": true` for before/after diff. |
132+
| 4-byte tracer | `"4byteTracer"` | Function selector frequency statistics. |
133+
| Flat call tracer | `"flatCallTracer"` | Parity-style flat list of all internal calls. |
134+
| Noop tracer | `"noopTracer"` | No output — useful for benchmarking execution time. |
135+
| Mux tracer | `"muxTracer"` | Run multiple tracers in a single pass. |
136+
137+
Custom JavaScript tracers are also supported.
138+
139+
### Parity-style (`trace_*`)
140+
141+
| Method | Description |
142+
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------ |
143+
| `trace_block` | Flat call traces for all transactions in a block. |
144+
| `trace_transaction` | Flat call traces for a single transaction. Returns `null` (not an error) when the transaction is not found, matching `mega-reth` behavior. |
145+
146+
Every method also has a `timed_`-prefixed alias (e.g. `timed_debug_traceBlockByNumber`) with identical behavior, letting callers tag client-side metrics separately.
147+
148+
## Command-line reference
149+
150+
Every flag has an equivalent environment variable.
151+
Command-line flags take precedence.
152+
153+
### Core flags
154+
155+
| Flag | Env variable | Required? | Description |
156+
| -------------------- | ------------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
157+
| `--addr` | `DEBUG_TRACE_SERVER_ADDR` | No | Listen address. Default: `0.0.0.0:8545`. |
158+
| `--rpc-endpoint` | `DEBUG_TRACE_SERVER_RPC_ENDPOINT` | Yes | MegaETH JSON-RPC endpoint(s) for block data. Comma-separated or repeated for failover (round-robin). |
159+
| `--witness-endpoint` | `DEBUG_TRACE_SERVER_WITNESS_ENDPOINT` | Yes | MegaETH endpoint(s) serving [`mega_getBlockWitness`](witness.md). Comma-separated or repeated for failover (primary-failover: first endpoint takes all traffic while healthy). |
160+
| `--genesis-file` | `DEBUG_TRACE_SERVER_GENESIS_FILE` | No | Path to genesis JSON. Uses the built-in chain spec when unset. |
161+
162+
### Local cache
163+
164+
Omit `--data-dir` to run in stateless mode (no local storage).
165+
166+
| Flag | Env variable | Default | Description |
167+
| ------------------------ | ----------------------------------------- | --------- | ----------------------------------------------------------------------------------------------------------------- |
168+
| `--data-dir` | `DEBUG_TRACE_SERVER_DATA_DIR` | _(unset)_ | Directory for the local database. Enables local cache mode when set. |
169+
| `--start-block` | `DEBUG_TRACE_SERVER_START_BLOCK` | _(unset)_ | Block **hash** for the initial anchor. When unset, the server fetches the latest block. Only used on first start. |
170+
| `--blocks-to-keep` | `DEBUG_TRACE_SERVER_BLOCKS_TO_KEEP` | `1000` | Recent blocks to retain. Older blocks are pruned. |
171+
| `--db-max-size` | `DEBUG_TRACE_SERVER_DB_MAX_SIZE` | `0` (off) | Max database file size (e.g. `10GB`). Triggers extra pruning (100-block batches) when exceeded. |
172+
| `--pruner-interval-secs` | `DEBUG_TRACE_SERVER_PRUNER_INTERVAL_SECS` | `300` | Seconds between pruning cycles. |
173+
174+
### Timeout and concurrency
175+
176+
| Flag | Env variable | Default | Description |
177+
| ----------------------------------- | ---------------------------------------------------- | --------- | ------------------------------------------------------------------------------------------------- |
178+
| `--witness-timeout` | `DEBUG_TRACE_SERVER_WITNESS_TIMEOUT` | `8` | Witness fetch timeout in seconds. |
179+
| `--block-fetch-timeout` | `DEBUG_TRACE_SERVER_BLOCK_FETCH_TIMEOUT_SECS` | `13` | Total timeout in seconds for all data needed for one block (header + witness + body + contracts). |
180+
| `--rpc-per-attempt-timeout-ms` | `DEBUG_TRACE_SERVER_RPC_PER_ATTEMPT_TIMEOUT_MS` | _(unset)_ | Per-attempt RPC timeout in milliseconds. Must be ≥ 100. |
181+
| `--data-max-concurrent-requests` | `DEBUG_TRACE_SERVER_DATA_MAX_CONCURRENT_REQUESTS` | unlimited | Cap on concurrent in-flight data requests (blocks, headers, code). |
182+
| `--witness-max-concurrent-requests` | `DEBUG_TRACE_SERVER_WITNESS_MAX_CONCURRENT_REQUESTS` | unlimited | Cap on concurrent in-flight witness fetches, independent of the data cap. |
183+
184+
{% hint style="info" %}
185+
The data and witness concurrency caps are independent semaphores — a burst on one path cannot starve the other.
186+
When using the public RPC, setting both to `4` prevents HTTP 429 rate-limiting.
187+
{% endhint %}
188+
189+
### Response cache
190+
191+
| Flag | Env variable | Default | Description |
192+
| ---------------------------------- | --------------------------------------------------- | ------- | ----------------------------------------------------------------------- |
193+
| `--response-cache-max-size` | `DEBUG_TRACE_SERVER_RESPONSE_CACHE_MAX_SIZE` | `1GB` | Maximum memory for cached responses. Accepts `KB`, `MB`, `GB` suffixes. |
194+
| `--response-cache-estimated-items` | `DEBUG_TRACE_SERVER_RESPONSE_CACHE_ESTIMATED_ITEMS` | `1000` | Initial capacity hint. Set to `0` to disable the cache entirely. |
195+
196+
The cache stores pre-serialized JSON responses keyed by `(block_number, tracer_type)`.
197+
Different tracers on the same block are cached independently.
198+
Entries are invalidated automatically on chain reorganization.
199+
200+
### Monitoring
201+
202+
| Flag | Env variable | Default | Description |
203+
| ------------------- | ------------------------------------ | ------- | --------------------------------------- |
204+
| `--metrics-enabled` | `DEBUG_TRACE_SERVER_METRICS_ENABLED` | `false` | Enable the Prometheus metrics endpoint. |
205+
| `--metrics-port` | `DEBUG_TRACE_SERVER_METRICS_PORT` | `9090` | Port for the metrics HTTP endpoint. |
206+
207+
When enabled, scrape `http://<HOST>:9090/metrics`.
208+
209+
| Metric | Type | What it tells you |
210+
| ----------------------------------------------------------------- | --------- | ------------------------------------------------ |
211+
| `debug_trace_rpc_requests_total` | Counter | Total RPC requests, labelled by method. |
212+
| `debug_trace_rpc_errors_total` | Counter | Total RPC errors, labelled by method. |
213+
| `debug_trace_request_duration_seconds` | Histogram | End-to-end request latency. |
214+
| `debug_trace_inflight_requests` | Gauge | Currently in-flight requests. |
215+
| `debug_trace_cache_hits_total` / `debug_trace_cache_misses_total` | Counter | Response cache hit/miss. |
216+
| `debug_trace_evm_execution_seconds` | Histogram | EVM trace execution time per request. |
217+
| `debug_trace_upstream_duration_seconds` | Histogram | Upstream RPC latency, labelled by method. |
218+
| `debug_trace_witness_bytes` | Histogram | Witness payload size. |
219+
| `debug_trace_local_chain_height` | Gauge | Latest block in the local database. |
220+
| `debug_trace_db_size_bytes` | Gauge | Database file size on disk. |
221+
| `debug_trace_reorg_depth` | Histogram | Chain reorg depth detected by the sync pipeline. |
222+
223+
### Logging
224+
225+
Logging shares the same `--log.*` flags and `STATELESS_LOG_*` environment variables as the [stateless validator](stateless-validation.md#logging-flags).
226+
See that page for the full flag table.
227+
228+
Recommended production settings:
229+
230+
```bash
231+
--log.stdout-format json \
232+
--log.stdout-filter info \
233+
--log.file-directory /var/log/dts \
234+
--log.file-filter debug \
235+
--log.file-max-size 500 \
236+
--log.file-max-files 10
237+
```
238+
239+
## Environment variables
240+
241+
All flags can be set via environment variables — convenient for container or orchestrator-based deployments:
242+
243+
```bash
244+
DEBUG_TRACE_SERVER_ADDR=0.0.0.0:8545
245+
DEBUG_TRACE_SERVER_RPC_ENDPOINT=https://mainnet.megaeth.com/rpc
246+
DEBUG_TRACE_SERVER_WITNESS_ENDPOINT=https://mainnet.megaeth.com/rpc
247+
DEBUG_TRACE_SERVER_DATA_DIR=/data/dts
248+
DEBUG_TRACE_SERVER_BLOCKS_TO_KEEP=5000
249+
DEBUG_TRACE_SERVER_RESPONSE_CACHE_MAX_SIZE=2GB
250+
DEBUG_TRACE_SERVER_METRICS_ENABLED=true
251+
STATELESS_LOG_STDOUT=info
252+
STATELESS_LOG_STDOUT_FORMAT=json
253+
```
254+
255+
## Troubleshooting
256+
257+
**Database needs a reset.**
258+
Stop the service, delete `trace_server.redb` from the data directory, and restart.
259+
The server re-syncs from the latest block automatically.
260+
261+
```bash
262+
rm /data/dts/trace_server.redb
263+
systemctl restart dts
264+
```
265+
266+
**Upstream RPC is unreachable.**
267+
The server retries with round-robin failover and exponential backoff.
268+
While all endpoints are down, the sync pipeline stalls (it does not crash) and trace requests return error `-32001` after `--block-fetch-timeout` expires.
269+
Service resumes automatically once any endpoint recovers.
270+
271+
**High memory usage.**
272+
Lower `--response-cache-max-size` if the response cache is consuming too much memory.
273+
The contract bytecode cache is process-lifetime — chains with many deployed contracts accumulate bytecode in memory over time.
274+
275+
**Slow trace requests.**
276+
Use Prometheus metrics to isolate the bottleneck:
277+
278+
- `debug_trace_upstream_duration_seconds` high — upstream RPC is slow.
279+
- `debug_trace_evm_execution_seconds` high — the block has many or complex transactions.
280+
- `debug_trace_witness_bytes` large — witness payloads are large, causing slow fetches.
281+
282+
The server logs a warning (`slow stages detected`) for any request stage exceeding 1 second.
283+
284+
## Related pages
285+
286+
- [Debugging Transactions](../dev/send-tx/debugging.md) — trace via the public RPC or `mega-evme` without running your own server
287+
- [Stateless Validation](stateless-validation.md) — the stateless validator that cryptographically verifies every block
288+
- [Get Block Witness](witness.md) — `mega_getBlockWitness` RPC reference and witness data layout
289+
- [debug-trace-server source](https://github.com/megaeth-labs/stateless-validator/tree/main/bin/debug-trace-server) — upstream Rust implementation

‎lychee.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ offline = true
99
root_dir = "docs"
1010

1111
# Check anchor fragments (#section-name)
12-
include_fragments = true
12+
include_fragments = "full"
1313

1414
# Skip these URL patterns (regex)
1515
exclude = [

0 commit comments

Comments
 (0)