Skip to content

Commit 810b8b6

Browse files
authored
docs: refresh CLAUDE.md, devnet skills, and Table docs to match code (#532)
## Why CLAUDE.md and the devnet skills had drifted from the code as PRs landed. Every claim below was verified against the current source before changing. ## What **CLAUDE.md** - Structure: 12 workspace crates — `net/api` and `common/test-fixtures` were missing from the tree; added the newer `blockchain/src` modules (`block_builder.rs`, `aggregation.rs`, `reaggregate.rs`, `sync_status.rs`) - Attestation pipeline: buffers are `new_payloads`/`known_payloads` (promotion at intervals 0/4), not `new_attestations`/`known_attestations` - Req/Resp: added `BlocksByRange` - Block-fetch retry: doubling from `INITIAL_BACKOFF_MS` (5ms → 2560ms), `MAX_FETCH_RETRIES` (10) attempts — was "10/40/160/640/2560, max 5" - XMSS signature size: `SIGNATURE_SIZE` = 2536 bytes, not 3112 - Dependencies: `ethereum_ssz`/`tree_hash` → `libssz`(+derive/types)/`libssz-merkle`; `tree_hash_root()` → `hash_tree_root()` - `stf_spectests.rs` lives in `crates/blockchain/state_transition/tests/`, not `crates/blockchain/tests/` - Replaced the rotted `store.rs:368` line reference with the function name (`on_gossip_attestation`) - Added a `docs/` index line to Resources **devnet-log-review skill** - The justification threshold examples were wrong: code is `3 * votes >= 2 * N` (= `ceil(2N/3)`), so 6 validators need 4 votes and 9 need 6 — the skill said 5 and 7 (">2/3") - Block proposal flow updated for the pre-build path (#445) and the `Finished building block` marker (#474) **test-pr-devnet skill** - Log analysis now points at this repo's own `devnet-log-review` skill (was pointing into lean-quickstart) and dumps container logs first - Fixed a broken relative link to CLAUDE.md; removed the stale client protocol-compatibility table **tables.rs** - `BlockSignatures` doc said `H256 -> BlockSignatures` and "all other blocks must have an entry"; the key is `(slot || root)` (slot-ordered pruning) and entries are pruned below the finalized boundary - `States` doc now references `SNAPSHOT_ANCHOR_INTERVAL` by name instead of hardcoding 1024 Docs/comments only — no behavior change.
1 parent 405a258 commit 810b8b6

4 files changed

Lines changed: 58 additions & 55 deletions

File tree

.claude/skills/devnet-log-review/SKILL.md

Lines changed: 14 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -145,7 +145,7 @@ grep "signature verification failed" lantern_0.log
145145

146146
### Finalization Debugging
147147

148-
Finalization should advance every 6-12 slots. If it stalls, investigate:
148+
On a healthy devnet finalization advances every few slots. If it stalls, investigate:
149149

150150
```bash
151151
# Check finalization progress
@@ -154,9 +154,11 @@ grep "finalized_slot=" ethlambda_0.log | tail -20
154154
# If finalized_slot stays same for 50+ slots → finalization stalled
155155
```
156156

157-
**Finalization requires >2/3 supermajority:**
158-
- 6 validators → need 5 votes minimum
159-
- 9 validators → need 7 votes minimum
157+
**Justification requires a ≥2/3 supermajority** (`3 * votes >= 2 * validator_count`,
158+
i.e. `ceil(2N/3)` — see `crates/blockchain/state_transition/src/lib.rs`):
159+
- 6 validators → need 4 votes minimum
160+
- 9 validators → need 6 votes minimum
161+
- 16 validators → need 11 votes minimum
160162

161163
**See [references/FINALIZATION_DEBUG.md](references/FINALIZATION_DEBUG.md) for:**
162164
- Common causes of finalization stalls
@@ -184,15 +186,15 @@ Different clients have different log formats and key patterns.
184186

185187
## Block Proposal Flow (ethlambda)
186188

187-
A healthy block proposal follows this sequence:
189+
Since the pre-build change (#445), the proposer builds at the *previous* slot's
190+
interval 4 and publishes aligned to the slot boundary. A healthy block proposal
191+
follows this sequence:
188192

189-
1. `We are the proposer for this slot` - Node detects it's the proposer
190-
2. `TODO precompute poseidons in parallel + SIMD` - XMSS aggregate proof starts
191-
3. `packed_pcs_commit` - Proof commitment
192-
4. `Logup data` - Logup protocol data
193-
5. `AIR proof{table=poseidon16}` / `AIR proof{table=poseidon24}` - AIR proofs
194-
6. `Published block` - Block successfully built and published
195-
7. `Published block to gossipsub` - Block broadcast to network
193+
1. `We are the proposer for this slot` - Node detects it's the proposer (fires one slot early)
194+
2. leanVM proving output (`packed_pcs_commit`, `Logup data`, `AIR proof{table=poseidon16|poseidon24}`) - XMSS aggregate proof
195+
3. `Finished building block` - Build complete (#474; this is where the build-time metric stops)
196+
4. `Published block` - Published at the slot boundary (the gap to step 3 is idle wait, not build cost)
197+
5. `Published block to gossipsub` - Block broadcast to network
196198

197199
## Summary Report Format
198200

.claude/skills/test-pr-devnet/SKILL.md

Lines changed: 7 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -53,7 +53,7 @@ Test ethlambda branch changes in a multi-client local devnet with zeam (Zig), re
5353
**Success criteria:**
5454
- ✅ No errors in ethlambda logs
5555
- ✅ All 4 nodes at same head slot
56-
- ✅ Finalization advancing (every 6-12 slots)
56+
- ✅ Finalization advancing (every few slots on a healthy devnet)
5757
- ✅ Each validator produces blocks for their slots
5858

5959
### Sync Recovery (~90-120s)
@@ -139,25 +139,13 @@ sleep 10 # Wait for sync
139139
# Quick status
140140
.claude/skills/test-pr-devnet/scripts/check-status.sh
141141
142-
# Detailed analysis (use devnet-log-review skill in lean-quickstart)
143-
cd $LEAN_QUICKSTART
142+
# Detailed analysis (use this repo's devnet-log-review skill; dump logs first)
143+
for node in zeam_0 ream_0 qlean_0 ethlambda_0; do
144+
docker logs "$node" > "${node}.log" 2>&1
145+
done
144146
.claude/skills/devnet-log-review/scripts/analyze-logs.sh
145147
```
146148
147-
## Protocol Compatibility
148-
149-
| Client | Status | Gossipsub | BlocksByRoot |
150-
|--------|--------|-----------|--------------|
151-
| ream | ✅ Full | ✅ Full | ✅ Full |
152-
| zeam | ✅ Full | ✅ Full | ⚠️ Limited |
153-
| qlean | ✅ Full | ✅ Full | ⚠️ Limited |
154-
| ethlambda | ✅ Full | ✅ Full | ✅ Full |
155-
156-
**Notes:**
157-
- zeam/qlean BlocksByRoot errors are expected (not a blocker)
158-
- ream ↔ ethlambda BlocksByRoot should work perfectly
159-
- All clients use Gossipsub for block propagation
160-
161149
## Verification Checklist
162150
163151
| Check | Command | Expected |
@@ -256,5 +244,5 @@ docker logs ethlambda_0 2>&1 | grep -i "peer\|connection" | head -20
256244
257245
## References
258246
259-
- **[ethlambda CLAUDE.md](../../CLAUDE.md)** - Development workflow, detailed debugging commands
260-
- **[lean-quickstart devnet-log-review](../../../lean-quickstart/.claude/skills/devnet-log-review/SKILL.md)** - Comprehensive log analysis
247+
- **[ethlambda CLAUDE.md](../../../CLAUDE.md)** - Development workflow, detailed debugging commands
248+
- **[devnet-log-review](../devnet-log-review/SKILL.md)** - Comprehensive log analysis (in this repo)

CLAUDE.md

Lines changed: 30 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ Not to be confused with Ethereum consensus clients AKA Beacon Chain clients AKA
99
**Rust version:** 1.92.0 (edition 2024)
1010
**Test fixtures release:** Download latest production fixtures from leanSpec releases
1111

12-
## Codebase Structure (10 crates)
12+
## Codebase Structure (12 workspace crates)
1313

1414
```
1515
bin/ethlambda/ # Entry point, CLI, orchestration
@@ -18,17 +18,24 @@ crates/
1818
blockchain/ # State machine actor (GenServer pattern)
1919
├─ src/lib.rs # BlockChain actor, tick events, validator duties
2020
├─ src/store.rs # Fork choice store, block/attestation processing
21+
├─ src/block_builder.rs # Block assembly (pre-built at previous slot's interval 4)
22+
├─ src/aggregation.rs # Interval-2 signature aggregation worker
23+
├─ src/reaggregate.rs # Re-aggregation of block-borne votes on import
24+
├─ src/sync_status.rs # Sync-gate tracker (suppresses duties while syncing)
2125
├─ src/key_manager.rs # Validator key management and signing
2226
├─ src/metrics.rs # Blockchain-level Prometheus metrics
23-
├─ fork_choice/ # LMD GHOST implementation (3SF-mini)
24-
└─ state_transition/ # STF: process_slots, process_block, attestations
27+
├─ fork_choice/ # [crate] LMD GHOST implementation (3SF-mini)
28+
└─ state_transition/ # [crate] STF: process_slots, process_block, attestations
29+
├─ src/justified_slots_ops.rs # Relative-index helpers for justified_slots
2530
└─ src/metrics.rs # State transition timing + counters
2631
common/
2732
├─ types/ # Core types (State, Block, Attestation, Checkpoint)
2833
├─ crypto/ # XMSS aggregation (leansig wrapper)
29-
└─ metrics/ # Prometheus re-exports, TimingGuard, gather utilities
34+
├─ metrics/ # Prometheus re-exports, TimingGuard, gather utilities
35+
└─ test-fixtures/ # Spec-fixture loading (prod dep of rpc's Hive test driver)
3036
net/
31-
├─ p2p/ # libp2p: gossipsub + req-resp (Status, BlocksByRoot)
37+
├─ api/ # Actor protocol traits wiring BlockChain ↔ P2P
38+
├─ p2p/ # libp2p: gossipsub + req-resp (Status, BlocksByRoot, BlocksByRange)
3239
│ ├─ src/gossipsub/ # Topic encoding, message handling
3340
│ ├─ src/req_resp/ # Request/response codec and handlers
3441
│ └─ src/metrics.rs # Peer connection/disconnection tracking
@@ -56,12 +63,14 @@ Interval 4: Accept accumulated attestations; build the NEXT slot's block and pub
5663

5764
### Attestation Pipeline
5865
```
59-
Gossip → Signature verification → new_attestations (pending)
60-
↓ (interval 4)
61-
promote → known_attestations (fork choice active)
66+
Gossip → Signature verification → new_payloads (pending)
67+
↓ (intervals 0/4)
68+
promote → known_payloads (fork choice active)
6269
6370
Fork choice head update
6471
```
72+
(Store buffer fields are `new_payloads`/`known_payloads`; the accessors are named
73+
`extract_latest_new_attestations`/`extract_latest_known_attestations`.)
6574

6675
### State Transition Phases
6776
1. **process_slots()**: Advance through empty slots, update historical roots
@@ -105,7 +114,7 @@ let byte: u8 = code.into();
105114

106115
### Ownership for Large Structures
107116
```rust
108-
// Prefer taking ownership to avoid cloning large data (signatures ~3KB)
117+
// Prefer taking ownership to avoid cloning large data (signatures ~2.5KB)
109118
pub fn insert_signed_block(&mut self, root: H256, signed_block: SignedBlock) { ... }
110119

111120
// Add .clone() at call site if needed - makes cost explicit
@@ -252,7 +261,7 @@ actual_slot = finalized_slot + 1 + relative_index
252261

253262
**XMSS (eXtended Merkle Signature Scheme):**
254263
- Post-quantum signature scheme
255-
- 52-byte public keys, 3112-byte signatures
264+
- 52-byte public keys, 2536-byte signatures (`SIGNATURE_SIZE` in `common/types/src/signature.rs`)
256265
- Epoch-based to prevent reuse
257266
- Aggregation via leanVM (previously leanMultisig) for efficiency
258267

@@ -268,11 +277,11 @@ actual_slot = finalized_slot + 1 + relative_index
268277
- Topic: `/leanconsensus/{fork_digest}/{block|aggregation|attestation_N}/ssz_snappy`
269278
- `fork_digest` is a 4-byte hex string (no `0x` prefix); currently the dummy `12345678` agreed across clients
270279
- Mesh size: 8 (6-12 bounds), heartbeat: 700ms
271-
- **Req/Resp**: Status, BlocksByRoot (snappy frame compression + varint length)
280+
- **Req/Resp**: Status, BlocksByRoot, BlocksByRange (snappy frame compression + varint length)
272281

273282
### Retry Strategy on Block Requests
274-
- Exponential backoff: 10ms, 40ms, 160ms, 640ms, 2560ms
275-
- Max 5 attempts, random peer selection on retry
283+
- Exponential backoff: doubling from `INITIAL_BACKOFF_MS` (5ms → 2560ms)
284+
- Max `MAX_FETCH_RETRIES` (10) attempts, random peer selection on retry
276285

277286
### Message IDs
278287
- 20-byte truncated SHA256 of: domain (valid/invalid snappy) + topic + data
@@ -301,9 +310,9 @@ GENESIS_VALIDATORS:
301310
### Test Categories
302311
1. **Unit tests**: Embedded in source files
303312
2. **Spec tests**: From `leanSpec/fixtures/consensus/`
304-
- `forkchoice_spectests.rs` (uses `on_block_without_verification`)
305-
- `signature_spectests.rs`
306-
- `stf_spectests.rs` (state transition)
313+
- `crates/blockchain/tests/forkchoice_spectests.rs` (uses `on_block_without_verification` via `spec_test_runner`)
314+
- `crates/blockchain/tests/signature_spectests.rs`
315+
- `crates/blockchain/state_transition/tests/stf_spectests.rs` (state transition)
307316

308317
### Running Tests
309318
```bash
@@ -316,7 +325,7 @@ cargo test -p ethlambda-blockchain --test forkchoice_spectests -- --test-threads
316325

317326
### Aggregator Flag Required for Finalization
318327
- At least one node **must** be started with `--is-aggregator` to finalize blocks
319-
- Without this flag, attestations pass signature verification and are logged as "Attestation processed", but the signature is never stored for aggregation (`store.rs:368`), so blocks are always built with `attestation_count=0`
328+
- Without this flag, attestations pass signature verification and are logged as "Attestation processed", but the signature is never stored for aggregation (the `is_aggregator` gate in `on_gossip_attestation`, `store.rs`), so blocks are always built with `attestation_count=0`
320329
- The attestation pipeline: gossip → verify signature → store gossip signature (only if `is_aggregator`) → aggregate at interval 2 → promote to known → pack into blocks
321330
- **Symptom**: `justified_slot=0` and `finalized_slot=0` indefinitely despite healthy block production and attestation gossip
322331

@@ -368,7 +377,7 @@ and are consumed during the tick pipeline (promotion at intervals 0/4,
368377
aggregation at interval 2).
369378

370379
### State Root Computation
371-
- Always computed via `tree_hash_root()` after full state transition
380+
- Always computed via `hash_tree_root()` after full state transition
372381
- Must match proposer's pre-computed `block.state_root`
373382

374383
### Finalization Checks
@@ -383,8 +392,8 @@ aggregation at interval 2).
383392

384393
**Critical:**
385394
- `leansig`: XMSS signatures (leanEthereum project)
386-
- `ethereum_ssz`: SSZ serialization
387-
- `tree_hash`: Merkle tree hashing
395+
- `libssz` / `libssz-derive` / `libssz-types`: SSZ serialization
396+
- `libssz-merkle`: Merkle tree hashing (`hash_tree_root()`)
388397
- `spawned-concurrency`: Actor model
389398
- `libp2p`: P2P networking (custom LambdaClass fork)
390399
- `vergen-git2`: Build-time git commit/branch info embedded in binary
@@ -397,6 +406,7 @@ aggregation at interval 2).
397406

398407
**Specs:** `leanSpec/src/lean_spec/` (Python reference implementation)
399408
**Devnet:** `lean-quickstart` (github.com/blockblaz/lean-quickstart)
409+
**Docs:** `docs/` — `rpc.md`, `metrics.md`, `checkpoint_sync.md`, `3sf_mini.md`, `lmd_ghost.md` (mdbook via `make docs`)
400410
**Releases:** See `RELEASE.md` for release process documentation
401411

402412
## Other implementations

crates/storage/src/api/tables.rs

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,15 +5,18 @@ pub enum Table {
55
BlockHeaders,
66
/// Block body storage: H256 -> BlockBody
77
BlockBodies,
8-
/// Block signatures storage: H256 -> BlockSignatures
8+
/// Block signatures storage: (slot || root) -> BlockSignatures
99
///
1010
/// Stored separately from blocks because the genesis block has no signatures.
11-
/// All other blocks must have an entry in this table.
11+
/// Keyed by slot || root so pruning can scan in slot order and stop early.
12+
/// Non-genesis blocks have an entry until finalized: signatures below the
13+
/// finalized boundary are pruned (`prune_old_block_signatures`), while
14+
/// headers and bodies are kept forever.
1215
BlockSignatures,
1316
/// State storage: H256 -> State
1417
///
15-
/// Holds full-state snapshots only: the bootstrap anchor plus one anchor per
16-
/// 1024-slot window. Never pruned. Non-anchor states live in `StateDiffs` and
18+
/// Holds full-state snapshots only: the bootstrap anchor plus one anchor
19+
/// every `SNAPSHOT_ANCHOR_INTERVAL` slots. Never pruned. Non-anchor states live in `StateDiffs` and
1720
/// are reconstructed on demand (memoized by an in-memory cache).
1821
States,
1922
/// State diffs: H256 -> StateDiff

0 commit comments

Comments
 (0)