Merge branch 'main' into sdhist

Author: AskAlexSharov

.claude/skills/erigondb-sync-integration-test-plan/SKILL.md

@@ -0,0 +1,118 @@
+---
+name: erigondb-sync-integration-test-plan
+description: Integration test plan for erigondb.toml settings resolution across 3 runtime scenarios (legacy, fresh+downloader, fresh+no-downloader)
+allowed-tools: Bash, Read
+user-invocable: false
+---
+
+# erigondb.toml Integration Test Plan
+
+Verify that `erigondb.toml` settings are correctly resolved across three distinct runtime scenarios: legacy datadirs, fresh sync with downloader, and fresh sync without downloader.
+
+Use this plan after any changes to the erigondb.toml resolution logic (creation, defaults, legacy detection, downloader delivery) to confirm all paths still work.
+
+## Parameters
+
+Two values must be provided by the caller:
+
+| Parameter | Description | Example |
+|-----------|-------------|---------|
+| **chain** | Chain name passed to `--chain=` | `hoodi`, `mainnet` |
+| **legacy datadir** | Path to an existing Erigon datadir that has `preverified.toml` but **no** `erigondb.toml` | `~/eth-nodes/erigon33-hoodi-stable` |
+
+All scenarios use `--prune.mode=archive`.
+
+## Pre-requisites
+
+1. **Built binary** — invoke `/erigon-build` if `./build/bin/erigon` does not exist.
+2. **Legacy datadir** — the user-provided path must contain `snapshots/preverified.toml` and must **not** contain `snapshots/erigondb.toml`.
+3. **Port availability** — use `/erigon-network-ports` for reference. Each scenario uses a different port offset (+100, +200, +300) via `/erigon-ephemeral` conflict detection.
+
+## Scenario A: Legacy datadir
+
+**Goal**: Starting on a legacy datadir (has `preverified.toml`, no `erigondb.toml`) creates `erigondb.toml` with legacy settings (`step_size = 1562500`).
+
+### Steps
+
+1. Create an ephemeral clone of the user's legacy datadir using `/erigon-ephemeral` (Mode B: clone). This gives an isolated copy so the original is untouched.
+2. Confirm pre-conditions on the clone:
+   - `snapshots/preverified.toml` exists
+   - `snapshots/erigondb.toml` does **not** exist
+3. Start erigon with port offset **+100** (via `/erigon-ephemeral` port conflict detection):
+   ```
+   ./build/bin/erigon \
+     --datadir=<clone-path> \
+     --chain=<chain> \
+     --prune.mode=archive \
+     <port flags at +100 offset>
+   ```
+4. Wait for startup (~15s). Check logs for a message indicating legacy settings detection (e.g., `Creating erigondb.toml with LEGACY settings`).
+5. Verify the generated file:
+   ```bash
+   cat <clone-path>/snapshots/erigondb.toml
+   ```
+   Expected: `step_size = 1562500`
+6. Kill the process and clean up the ephemeral datadir.
+
+## Scenario B: Fresh sync with downloader
+
+**Goal**: A fresh datadir (no `preverified.toml`, no `erigondb.toml`) starts with code defaults (`step_size = 1562500`), then the downloader delivers the network's `erigondb.toml` during the header-chain phase, which may have different settings (e.g., `step_size = 390625` for hoodi).
+
+### Steps
+
+1. Create an empty ephemeral datadir using `/erigon-ephemeral` (Mode A: empty).
+2. Start erigon with port offset **+200**:
+   ```
+   ./build/bin/erigon \
+     --datadir=<empty-path> \
+     --chain=<chain> \
+     --prune.mode=archive \
+     <port flags at +200 offset>
+   ```
+3. Check early logs for a message indicating defaults are being used (e.g., `erigondb.toml not found, using defaults`) with `step_size=1562500`.
+4. Wait for the header-chain download phase to complete (~30-120s depending on network). Watch logs for:
+   - `[1/6 OtterSync] Downloader completed header-chain`
+   - `Reading DB settings from existing erigondb.toml`
+   - `erigondb stepSize changed, propagating` (if the network's settings differ from the code defaults)
+5. Verify the delivered file:
+   ```bash
+   cat <empty-path>/snapshots/erigondb.toml
+   ```
+   The values come from the network's published `erigondb.toml` (check the chain's webseed for the canonical values).
+6. Kill the process and clean up the ephemeral datadir.
+
+## Scenario C: Fresh sync with `--no-downloader`
+
+**Goal**: A fresh datadir with `--no-downloader` immediately writes `erigondb.toml` with code defaults (`step_size = 1562500`).
+
+### Steps
+
+1. Create an empty ephemeral datadir using `/erigon-ephemeral` (Mode A: empty).
+2. Start erigon with port offset **+300** and `--no-downloader`:
+   ```
+   ./build/bin/erigon \
+     --datadir=<empty-path> \
+     --chain=<chain> \
+     --prune.mode=archive \
+     --no-downloader \
+     <port flags at +300 offset>
+   ```
+3. Check logs for a message indicating defaults were written immediately (e.g., `Initializing erigondb.toml with DEFAULT settings`) with `step_size=1562500`.
+4. Verify the file exists immediately:
+   ```bash
+   cat <empty-path>/snapshots/erigondb.toml
+   ```
+   Expected: `step_size = 1562500` (code defaults, since no downloader to provide network settings)
+5. Kill the process and clean up the ephemeral datadir.
+
+## Success Criteria
+
+| Scenario | Condition | Expected step_size | File timing |
+|----------|-----------|-------------------|-------------|
+| A (legacy) | `Creating erigondb.toml with LEGACY settings` log | 1,562,500 | Written immediately on startup |
+| B (fresh+downloader) | `erigondb.toml not found, using defaults` then `erigondb stepSize changed, propagating` | Code default 1,562,500 then network value (chain-dependent) | After downloader delivers it |
+| C (fresh+no-downloader) | `Initializing erigondb.toml with DEFAULT settings (nodownloader)` log | 1,562,500 | Written immediately on startup |
+
+## Cleanup
+
+After all scenarios complete, ensure all ephemeral datadirs and processes are cleaned up. Use `/erigon-ephemeral` Step 5 (cleanup) for each instance, or Step 6 (leftover detection) to find any stragglers.

.claude/skills/launch-bal-devnet-2/SKILL.md

@@ -0,0 +1,185 @@
+---
+name: launch-bal-devnet-2
+description: Launch erigon + Lighthouse on the bal-devnet-2 ethpandaops devnet (EIP-7928 Block Access Lists). Manages start/stop of both EL and CL clients with proper port offsets and JWT auth.
+allowed-tools: Bash, Read, Write, Edit, Glob
+allowed-prompts:
+  - tool: Bash
+    prompt: start, stop, and manage erigon and lighthouse processes for bal-devnet-2
+---
+
+# Launch bal-devnet-2 (EIP-7928 BAL Devnet)
+
+Run erigon (EL) + Lighthouse (CL) on the bal-devnet-2 ethpandaops devnet for testing EIP-7928 Block Access Lists.
+
+## Network Details
+
+| Parameter | Value |
+|-----------|-------|
+| Chain ID | 7033429093 |
+| Genesis timestamp | 1770388190 |
+| Amsterdam timestamp | 1770400508 (epoch 32) |
+| Lighthouse image | `ethpandaops/lighthouse:bal-devnet-2-65bb283` |
+| Lighthouse version | v8.0.1 (commit 65bb283, branch bal-devnet-2) |
+| Explorer | https://explorer.bal-devnet-2.ethpandaops.io |
+| Faucet | https://faucet.bal-devnet-2.ethpandaops.io |
+| RPC | https://rpc.bal-devnet-2.ethpandaops.io |
+| Checkpoint sync | https://checkpoint-sync.bal-devnet-2.ethpandaops.io |
+
+## Working Directory
+
+Ask the user where they want the working directory. Default suggestion: `~/bal-devnet-2/`.
+Use `$WORKDIR` throughout to refer to the chosen path.
+
+```
+$WORKDIR/
+├── genesis.json              # EL genesis
+├── config.yaml               # CL beacon config
+├── genesis.ssz               # CL genesis state
+├── testnet-config/           # Lighthouse testnet dir (config.yaml + genesis.ssz + deploy files)
+├── start-erigon.sh           # Erigon start script (run FIRST)
+├── start-lighthouse.sh       # Lighthouse start script (run SECOND)
+├── stop.sh                   # Stop both erigon + Lighthouse
+├── clean.sh                  # Stop, wipe data, re-init genesis
+├── erigon-data/              # Erigon datadir (contains jwt.hex)
+├── lighthouse-data/          # Lighthouse datadir
+├── erigon-console.log        # Erigon stdout/stderr
+└── lighthouse-console.log    # Lighthouse stdout/stderr
+```
+
+## Port Assignments (offset +100)
+
+| Service | Port | Protocol |
+|---------|------|----------|
+| Erigon HTTP RPC | 8645 | TCP |
+| Erigon Engine API (authrpc) | 8651 | TCP |
+| Erigon WebSocket | 8646 | TCP |
+| Erigon P2P | 30403 | TCP+UDP |
+| Erigon gRPC | 9190 | TCP |
+| Erigon Torrent | 42169 | TCP+UDP |
+| Erigon pprof | 6160 | TCP |
+| Erigon metrics | 6161 | TCP |
+| Lighthouse P2P | 9100 | TCP+UDP |
+| Lighthouse QUIC | 9101 | UDP |
+| Lighthouse HTTP API | 5152 | TCP |
+| Lighthouse metrics | 5264 | TCP |
+
+## Workflow
+
+### Step 1: Check Prerequisites
+
+1. Verify erigon binary exists at `./build/bin/erigon`. If not, invoke `/erigon-build`.
+2. Verify the Lighthouse Docker image is available:
+   ```bash
+   docker image inspect ethpandaops/lighthouse:bal-devnet-2-65bb283 > /dev/null 2>&1
+   ```
+   If not, pull it:
+   ```bash
+   docker pull ethpandaops/lighthouse:bal-devnet-2-65bb283
+   ```
+3. Verify config files exist in `$WORKDIR` (genesis.json, testnet-config/).
+   If not, download them:
+   ```bash
+   mkdir -p $WORKDIR/testnet-config
+   curl -sL -o $WORKDIR/genesis.json https://config.bal-devnet-2.ethpandaops.io/el/genesis.json
+   curl -sL -o $WORKDIR/testnet-config/config.yaml https://config.bal-devnet-2.ethpandaops.io/cl/config.yaml
+   curl -sL -o $WORKDIR/testnet-config/genesis.ssz https://config.bal-devnet-2.ethpandaops.io/cl/genesis.ssz
+   echo "0" > $WORKDIR/testnet-config/deposit_contract_block.txt
+   echo "0" > $WORKDIR/testnet-config/deploy_block.txt
+   ```
+
+### Step 2: Initialize Datadir (first run only)
+
+If `$WORKDIR/erigon-data/chaindata` does not exist:
+```bash
+./build/bin/erigon init --datadir $WORKDIR/erigon-data $WORKDIR/genesis.json
+```
+
+### Step 3: Create Scripts (first run only)
+
+If the start/stop/clean scripts don't exist yet, generate them. The scripts must use absolute paths based on `$WORKDIR`. Key details:
+
+**start-erigon.sh** — Runs erigon with `--externalcl`. Must start FIRST (creates JWT secret).
+- Env vars: `ERIGON_EXEC3_PARALLEL=true`, `ERIGON_ASSERT=true`, `ERIGON_EXEC3_WORKERS=12`, `LOG_HASH_MISMATCH_REASON=true`
+- Flags: `--datadir=$WORKDIR/erigon-data`, `--externalcl`, `--networkid=7033429093`, all 16 EL bootnodes, erigon static peers, `--prune.mode=minimal`, all offset ports (see port table), `--http.api=eth,erigon,engine,debug`, `--pprof`, `--metrics`
+- EL bootnodes: fetch from `https://config.bal-devnet-2.ethpandaops.io/api/v1/nodes/inventory` (extract enode URLs from `execution.enode` fields)
+
+**start-lighthouse.sh** — Runs Lighthouse via Docker with `--network=host`. Must start SECOND.
+- Checks JWT exists at `$WORKDIR/erigon-data/jwt.hex`
+- Docker container name: `bal-devnet-2-lighthouse`
+- Mounts: `$WORKDIR/testnet-config:/config:ro`, `$WORKDIR/lighthouse-data:/data`, JWT as `/jwt.hex:ro`
+- Flags: `--testnet-dir=/config`, `--execution-endpoint=http://127.0.0.1:8651`, `--execution-jwt=/jwt.hex`, all 15 CL ENR bootnodes, offset ports, `--checkpoint-sync-url=https://checkpoint-sync.bal-devnet-2.ethpandaops.io`
+- CL bootnodes: fetch from same inventory URL (extract ENR entries from `consensus.enr` fields)
+
+**stop.sh** — Stops Lighthouse (`docker stop bal-devnet-2-lighthouse`) then erigon (`pkill -f "datadir.*bal-devnet-2/erigon-data"`).
+
+**clean.sh** — Runs `stop.sh`, removes erigon chain data (chaindata, snapshots, txpool, nodes, temp) and lighthouse data, re-initializes genesis.
+
+### Step 4: Start Erigon (FIRST)
+
+Erigon must start first because it creates the JWT secret that Lighthouse needs.
+
+```bash
+cd $WORKDIR && nohup bash start-erigon.sh > erigon-console.log 2>&1 &
+```
+
+Verify it started:
+- Check `tail $WORKDIR/erigon-console.log` for startup messages
+- Check JWT exists: `ls $WORKDIR/erigon-data/jwt.hex`
+- Check port binding: `ss -tlnp | grep 8651`
+
+### Step 5: Start Lighthouse (SECOND)
+
+After erigon is running and JWT exists:
+
+```bash
+cd $WORKDIR && nohup bash start-lighthouse.sh > lighthouse-console.log 2>&1 &
+```
+
+Verify it started:
+- Check `tail $WORKDIR/lighthouse-console.log` for "Lighthouse started"
+- Look for "Loaded checkpoint block and state" (checkpoint sync)
+- Look for `peers: "N"` showing peer connections
+
+### Step 6: Monitor
+
+```bash
+# Erigon sync progress
+tail -f $WORKDIR/erigon-console.log
+
+# Lighthouse sync progress
+tail -f $WORKDIR/lighthouse-console.log
+
+# Check erigon block height via RPC
+curl -s http://localhost:8645 -X POST -H "Content-Type: application/json" \
+  -d '{"jsonrpc":"2.0","method":"eth_blockNumber","params":[],"id":1}' | python3 -m json.tool
+
+# Check lighthouse sync status
+curl -s http://localhost:5152/eth/v1/node/syncing | python3 -m json.tool
+```
+
+### Step 7: Stop
+
+```bash
+bash $WORKDIR/stop.sh
+```
+
+This stops Lighthouse (via `docker stop`) then erigon (via `pkill`).
+
+### Step 8: Clean (wipe data and re-init)
+
+```bash
+bash $WORKDIR/clean.sh
+```
+
+This runs `stop.sh`, removes erigon chain data (chaindata, snapshots, txpool, nodes, temp) and lighthouse data, then re-initializes genesis. After clean, start again with Steps 4-5.
+
+## Troubleshooting
+
+| Problem | Solution |
+|---------|----------|
+| JWT auth fails | Ensure erigon started first and `jwt.hex` exists. Lighthouse must mount the same file. |
+| No EL peers | Check firewall allows port 30403. Try adding `--nat=extip:<your-ip>`. |
+| No CL peers | Check firewall allows port 9100/9101. ENR bootnodes may have changed — re-fetch from inventory. |
+| "Head is optimistic" | Normal during initial sync. Erigon is behind Lighthouse. Will resolve as erigon catches up. |
+| Engine API timeout | Check erigon is running and authrpc port 8651 is accessible. |
+| Port conflict | Check `ss -tlnp | grep <port>`. Kill conflicting process or use higher offset. |

.github/workflows/qa-constrained-tip-tracking.yml

@@ -22,7 +22,7 @@ jobs:
     runs-on: [ self-hosted, qa, "${{ matrix.backend }}", tip-tracking ]
     timeout-minutes: 780 # 13 hours
     env:
-      ERIGON_REFERENCE_DATA_DIR: /opt/erigon-versions/reference-version-main/datadir
+      ERIGON_REFERENCE_DATA_DIR: /opt/erigon-versions/reference-version/datadir
       ERIGON_TESTBED_DATA_DIR: /opt/erigon-testbed/datadir
       ERIGON_QA_PATH: /home/qarunner/erigon-qa
       TRACKING_TIME_SECONDS: 7200 # 2 hours

.github/workflows/qa-rpc-integration-tests-gnosis.yml

@@ -29,7 +29,6 @@ jobs:
       cancel-in-progress: true
     runs-on: [ self-hosted, qa, Gnosis, rpc-integration ]
     env:
-      ERIGON_REFERENCE_DATA_DIR: /opt/erigon-versions/gnosis-reference-version/datadir
       ERIGON_TESTBED_AREA: /opt/erigon-testbed
       ERIGON_QA_PATH: /home/qarunner/erigon-qa
       ERIGON_ASSERT: true
@@ -40,6 +39,21 @@ jobs:
       - name: Check out repository
         uses: actions/checkout@v6
 
+      - name: Set reference data dir based on branch
+        run: |
+          # For pull_request events base_ref is the target branch name; for push/dispatch parse from ref.
+          BRANCH="${{ github.base_ref }}"
+          if [ -z "$BRANCH" ]; then
+            BRANCH="${{ github.ref }}"
+            BRANCH="${BRANCH#refs/heads/}"
+          fi
+          if [[ "$BRANCH" == release/* ]]; then
+            VERSION="${BRANCH#release/}"
+            echo "ERIGON_REFERENCE_DATA_DIR=/opt/erigon-versions/gnosis-reference-version-${VERSION}/datadir" >> $GITHUB_ENV
+          else
+            echo "ERIGON_REFERENCE_DATA_DIR=/opt/erigon-versions/gnosis-reference-version/datadir" >> $GITHUB_ENV
+          fi
+
       - name: Clean Erigon Build Directory
         run: |
           make clean