docs: add stateless validator guide

[flyq]

Summary

Adds a new Stateless Validator documentation layer at docs/stateless/, targeting node operators running the upstream stateless-validator Rust client.

Changes

• New page: docs/stateless/stateless-validator.md — end-to-end operator reference, covering:

  • Motivation (independent verification, SALT witnesses, low hardware cost, auditable TCB)
  • Installation from source
  • Quick start (first run anchoring with --genesis-file / --start-block; subsequent runs)
  • CLI flag and environment variable reference, mirrored from upstream clap definitions
  • Log configuration (LogArgs: filters, formats, rotation)
  • Prometheus metric reference from metrics::names
  • Background deployment patterns (systemd, Docker)
  • Trust model and pairing with op-node / replica nodes
  • Troubleshooting common failure modes (lag, reorgs, RPC errors)

• New layer rules: docs/stateless/AGENTS.md — writing conventions for the stateless layer (reference tone, copy-pasteable commands, placeholder style, verification-against-source requirement before editing flag/metric tables).

• docs/SUMMARY.md — adds the "Stateless Validator" section so GitBook navigation picks up the new page.

• AGENTS.md (root) — registers the new layer in the architecture table and the project tree.

• docs/AGENTS.md — adds stateless/ to the list of layers that use the terse reference tone.

Test plan

• Spot-check CLI flag / env var / metric tables against upstream stateless-validator source

[claude]

Review

Good addition — well-structured, thorough CLI reference, and the metrics table is genuinely useful. A few things to address before merging.

Missing AGENTS.md for the stateless/ layer
docs/AGENTS.md says each layer has its own AGENTS.md with layer-specific writing rules, and CLAUDE.md repeats this. The new docs/stateless/ directory doesn't have one. Either add a minimal AGENTS.md (audience: node operators; tone: terse reference, same as dev layer) or document why this layer shares conventions with an existing one.

Four inline comments on specific lines — see the diff for details:

• Line 60: genesis file link points to the org root, not a usable download location
• Line 62: e.g. → e.g.,
• Line 178: start_validator.sh always passes --genesis-file but the prose says to omit it on subsequent runs — needs clarification or a conditional
• Line 279: ## Related Pages → ## Related pages (sentence case)

[claude]

Good overall — well-structured, covers the full operator lifecycle, and the CLI flag tables are thorough. A few issues to resolve before merging:

4 existing threads still open (not yet addressed):

• ## Related Pages → sentence case (## Related pages)
• Missing comma: e.g. → e.g.,
• Background launch script passes --genesis-file unconditionally, but subsequent runs should omit it
• Genesis file link points to GitHub org root (https://github.com/megaeth-labs) — needs a specific repo/file

2 new issues (inline comments added):

• Prometheus example output (lines 191–193): the #-prefixed lines shown are Prometheus metadata, not gauge values — the actual grep output would show bare metric_name value lines
• Terminology (line 264): Mini-Blocks → mini-blocks per docs/AGENTS.md terminology table

[claude]

Review

The stateless validator guide is well-structured and thorough — the systemd deployment section, flag/metric tables, and troubleshooting entries are exactly right for this audience.

Must fix before merge:

• temp.md at repo root must be deleted. It's a stale scratch file that should not ship. It also uses the legacy STATELESS_VALIDATOR_LOG_* env var names that the guide itself says are deprecated, so it would actively mislead operators who copy from the wrong place.

Prior threads still unresolved — please address:

1. Prometheus output example is wrong (thread open on line 243) — the #-prefixed lines in the curl output block are HELP/TYPE metadata, not values; the actual gauge lines have no # prefix. The suggestion in the thread is correct.

2. STATELESS_VALIDATOR_GENESIS_FILE in the systemd env file (thread on genesis/start-block) — the env file sets this on every run, but the "Subsequent runs" section says to omit --genesis-file once the DB is initialised. Either confirm the binary silently ignores it on re-runs (and add a note), or remove it from the env file and document the one-time ExecStartPre bootstrap pattern.

3. Heading case — ## Related Pages → ## Related pages (sentence case per docs/AGENTS.md).

4. Missing comma — e.g. copied → e.g., copied.

5. mini-blocks link — the Related Pages section was previously flagged as missing a [mini-blocks](../mini-block.md) entry; the term was removed but the link was never added.

Resolved this round: genesis file link (now correctly points to the specific file in the repo) and systemd example request (now fully addressed).

[claude]

Good progress — the systemd rewrite and temp.md removal clean up the main structural concerns from the last round.

Two remaining items:

1. Open thread still unresolved — ## Related Pages (line 357) needs sentence case: ## Related pages. That thread has been open since the first review pass.

2. mini-block link was dropped rather than fixed — the previous review flagged Mini-blocks capitalisation; the response was to remove the link entirely. The cross-reference to mini-block.md is genuinely useful for operators who want to understand what block types the validator re-executes. Please restore it with the correct casing:

   - [mini-blocks](../mini-block.md) — the two block types the validator re-executes
   

[claude]

Review

Solid, well-structured guide — covers the full operator lifecycle from first-run anchoring through systemd hardening and monitoring. The layer AGENTS.md is a good addition and the flag/env-var tables are clear.

Resolved stale threads — the three open threads (heading case, trailing backslash, e.g., comma) are all fixed in the current file; resolved them above.

One remaining issue — see inline comment on the monitoring section: the grep example output omits the # HELP / # TYPE metadata lines that Prometheus includes and the current pattern would match. A grep -v '^#' stage fixes it.

Minor suggestion — the [mini-blocks](../mini-block.md) cross-link was present in an earlier revision and appears to have been dropped when the capitalization thread was resolved. The page exists at docs/mini-block.md and is directly relevant to operators (validators re-execute mini-blocks). Worth adding back.

Otherwise this is ready.

[claude]

Review: LGTM with no blocking issues

All previous threads have been addressed and resolved.

What's here:
The guide covers the full operator journey end-to-end: installation from source, first-run anchoring, subsequent restarts, systemd unit, monitoring via Prometheus, trust model, and troubleshooting. The layer AGENTS.md is well-structured and enforces the verify-against-source discipline for the flag/metric tables.

Specifically confirmed:

• The grep -v '^#' Prometheus fix is in the current diff — the output example now accurately reflects pipe behavior. (Thread resolved.)
• ../architecture.md and ../mini-block.md both exist in docs/, so all relative links are valid.
• One-sentence-per-line convention is followed throughout the prose.
• SUMMARY.md, root AGENTS.md, and docs/AGENTS.md are all updated consistently.
• No stale temp.md scratch file remains.

Ready to merge once the requested reviewers sign off.

[claude]

All 18 previous review threads are resolved — the doc is in solid shape. One remaining inconsistency flagged inline: the Prometheus example output and the "During initial catch-up" explanation still carry the old 0xc0ffee anchor numbers (12.6M local, 14M remote, 1.4M lag). Since the Quick Start now anchors at finalized, a user following the guide would see lag near 0 immediately, not 1.3M. Suggestion and fix are in the inline comment.

[claude]

All previous review threads are resolved. One item remains before merging:

Open thread (monitoring section): The example output in the grep block still shows the old 0xc0ffee-era numbers (local 12.6M, remote 14M, lag 1.3M), and the "During initial catch-up" bullet still calls those out explicitly. Since the Quick Start now anchors at the finalized block (near tip), a fresh operator will see lag ≈ 0–10 immediately — the stale large-lag example is now misleading. The unresolved thread has a ready-made fix.

Otherwise the guide is solid: copy-pasteable commands are correct, the systemd pattern is clean, rate-limit caps are in place for the public RPC, and the flag/metric tables match upstream source.

[claude]

Review

This is a high-quality, well-structured addition. The previous review round was thorough and nearly all issues have been addressed — copy-pasteable commands are correct, the systemd deployment is production-grade, the metrics table is comprehensive, and the one-sentence-per-line rule is followed throughout.

One open thread still needs resolution: the monitoring example output.

Thread PRRT_kwDOODfgX859wFvO (on the Checking validation progress section) is unresolved. The example currently shows:

stateless_validator_validation_lag  1832

But the Quick Start now anchors at the finalized block (near tip), so a user following the guide as written will see validation_lag ≈ 0–10 almost immediately — not 1832. Showing a mid-catch-up state as the representative example is misleading and contradicts the updated "During initial catch-up (only if you anchored at an older block)" qualifier right below it.

The fix from the existing thread works: update the example numbers to show near-tip operation (lag ≈ 6), and drop or generalise the hardcoded "12.6M / 14M / ~1.4M" phrasing in the bullets. The surrounding explanation about tip_buffer = 3 and steady-state behaviour is already good and doesn't need to change.

Everything else is ready to merge once this is addressed.

[claude]

Review

All previous threads have been addressed except for one — the monitoring section example output. Two small items remain.

Remaining issues

1. Stale monitoring example (unresolved thread, now also flagged inline on lines 339–343): The validation_lag 1832 example output conflicts with the Quick Start anchor workflow. Since the guide now anchors at eth_getHeaderByNumber("finalized"), a first-time operator sees lag ≈ 3–5 immediately. The example should reflect steady-state operation.

2. Stale warning text (new, inline on line 225): The Bootstrap hint says "The value shown is only an example for the quick-start walkthrough" — but the command shows <YOUR_TRUSTED_BLOCK_HASH> (a placeholder), not a concrete value. Leftover from when 0xc0ffee was the example anchor.

Everything else

The rest of the guide is in good shape: flags/env vars/metric tables match upstream source, systemd unit includes rate-limit caps, log tail commands use the correct env var, and one-sentence-per-line formatting is clean throughout.

[claude]

All previous review threads are now addressed. The latest iteration looks ready:

• Monitoring example output updated to show near-zero lag (5 blocks) consistent with the finalized-anchor Quick Start workflow.
• "During initial catch-up" bullet correctly scoped to users who anchored at an older block, with no stale hardcoded block numbers.
• Stale 0xc0ffee reference in the systemd bootstrap hint is gone.
• Concurrency caps added in both Quick Start and the systemd env file.
• Prometheus grep command correctly strips metadata lines with grep -v '^#'.
• docs/architecture.md exists; cross-links are valid.
• temp.md removed.
• SUMMARY.md uses the suggested "Node Operator" section and "Stateless Validation" link text.

No new issues found. LGTM.

[claude]

All previous threads resolved. Fresh pass on the latest revision finds nothing new to flag:

• Monitoring example shows near-zero lag (5 blocks), consistent with the finalized-anchor Quick Start workflow.
• Rate-limit caps present in both Quick Start and the systemd env file.
• Prometheus grep -v '^#' filter in place.
• Log tail uses correct $STATELESS_LOG_FILE_DIRECTORY env var.
• Cross-link ../architecture.md is valid (file exists).
• One-sentence-per-line formatting is clean throughout.

LGTM.

[claude]

All 26 previous threads are resolved. One new issue flagged inline (broken sentence in the monitoring section). Otherwise this is in good shape — the Quick Start is accurate and safe, flag/env-var tables are consistent with source, Prometheus examples use grep -v '^#', rate-limiting guidance is present, and the SUMMARY structure is correct. Ready to merge after the grammar fix.

[claude]

All prior review threads are resolved. Reviewed the latest diff — no remaining issues.

The document is accurate, well-structured, and follows the layer's conventions:

• Quick start flow is correct: fetches the finalized header, anchors near-tip, concurrency caps included
• Prometheus grep correctly strips metadata lines with grep -v '^#'
• Monitoring example output matches near-tip anchor (lag 5, not 1.4M)
• One-sentence-per-line rule followed throughout
• Flag/env-var/metric tables mirror upstream source
• ../architecture.md link resolves — file exists

Ready to merge.

[claude]

All 24 prior threads are resolved. The document is technically accurate, follows conventions, and the Quick Start is copy-paste safe after previous rounds of live testing.

One remaining nit (not a blocker): the tail command in the Logs section expands incorrectly for users who followed the Quick Start using the CLI flag form:

tail -f "$STATELESS_LOG_FILE_DIRECTORY/stateless-validator.log"

If the user passed --log.file-directory ./validator-data (as shown in Quick Start) without also exporting the env var, $STATELESS_LOG_FILE_DIRECTORY is unset and the command silently becomes tail -f '/stateless-validator.log'. Using a hardcoded path consistent with the Quick Start example would be safer:

tail -f ./validator-data/stateless-validator.log

Worth fixing before merge, but fine to address as a follow-up if the PR is otherwise ready to ship.

[claude]

This is a thorough, well-structured operator reference. 24 of the 25 prior review threads have been resolved. One issue remains open and needs a fix before merge.

Remaining issue — tail command in the Logs section (line 273)

The open thread on tail -f "$STATELESS_LOG_FILE_DIRECTORY/stateless-validator.log" is correct: if an operator used the CLI flag --log.file-directory ./validator-data/logs (as the Quick Start shows), the env var $STATELESS_LOG_FILE_DIRECTORY is not set, so the command silently expands to tail -f /stateless-validator.log (or similar broken path).

The fix: align it with the hardcoded tail command already in the Quick Start (line 102):

tail -f ./validator-data/logs/stateless-validator.log

Note: the existing thread's suggestion (./validator-data/stateless-validator.log) is missing the logs/ subdirectory — the correct path is ./validator-data/logs/stateless-validator.log, matching the Quick Start's --log.file-directory ./validator-data/logs.

No other issues found in the current revision.

[claude]

All 27 review threads are now resolved.

The last open thread flagged $STATELESS_LOG_FILE_DIRECTORY as unexpanded in the Logs section — the current file already has a hardcoded path (./validator-data/logs/stateless-validator.log) consistent with the Quick Start, so that thread has been resolved.

Fresh pass on the latest revision — no new issues:

• Quick Start anchors at finalized, concurrency caps in place, log tail path hardcoded
• Prometheus grep -v '^#' filter present; example output shows near-zero lag (5 blocks)
• Flag/env-var/metric tables verified against upstream source
• Subsequent-run command omits --genesis-file and --start-block correctly
• SUMMARY uses Node Operation section header and Stateless Validation link text
• One-sentence-per-line rule followed throughout

LGTM — ready to merge.