Add hive test skill (#19615)

mh0lt · claude · web-flow · commit 3ce01bcd372b · 2026-03-04T17:15:51.000Z
This adds a claude skill to run hive tests, this allows a local agent to
set-up and run the suite of hive tests we use for development to confim
local changes don't cause regressions.

claude can pretty much work out how to use hive on its own, but this
skill adds a bit of scope and control to the process and also tells
claude how to set-up ephemeral instances from branches and set-up tests
on local code.

---------

Co-authored-by: Claude Opus 4.6 &lt;noreply@anthropic.com&gt;
diff --git a/.claude/skills/hive-test/SKILL.md b/.claude/skills/hive-test/SKILL.md
@@ -0,0 +1,298 @@
+# Skill: hive-test
+
+Run Ethereum Hive integration tests against a local Erigon build. Works from a clean
+environment -- no pre-existing hive installation required.
+
+## Metadata
+- user-invocable: true
+- description: Run Hive integration tests (engine, rpc, eest) against local Erigon
+- allowed-tools: Bash, Read, Write, Edit, Glob, Grep
+
+## Overview
+
+This skill sets up an ephemeral Hive test environment, builds a local Erigon Docker
+image, runs the requested test suites, reports results, and cleans up containers.
+
+## Arguments
+
+The user may specify one or more test suites in any combination:
+
+### Individual suites
+| Suite name | Simulator | Description |
+|-----------|-----------|-------------|
+| `exchange-capabilities` | ethereum/engine | Engine exchange-capabilities |
+| `withdrawals` | ethereum/engine | Engine withdrawals |
+| `cancun` | ethereum/engine | Engine cancun |
+| `api` | ethereum/engine | Engine API |
+| `auth` | ethereum/engine | Engine auth |
+| `rpc-compat` | ethereum/rpc | RPC compatibility |
+| `eest` | ethereum/eels/consume-engine | Execution Spec Tests (version auto-discovered) |
+| `eest-bal` | ethereum/eels/consume-engine | EEST BAL amsterdam fixtures (version auto-discovered) |
+
+### Groups
+| Group name | Expands to |
+|-----------|------------|
+| `engine` | exchange-capabilities, withdrawals, cancun, api, auth |
+| `all` | Every suite listed above |
+
+### Examples
+- `/hive-test api` - Run just the engine API suite
+- `/hive-test withdrawals api` - Run withdrawals and API suites
+- `/hive-test engine` - Run all engine suites
+- `/hive-test engine rpc-compat` - Run all engine suites plus rpc-compat
+- `/hive-test eest-bal` - Run BAL-specific EEST tests
+- `/hive-test all` - Run everything
+
+### Options
+- **erigon-path** - Path to local erigon source (default: current working directory)
+- **branch=BRANCH** - Clone erigon from a remote branch instead of using the local
+  working directory. The branch is cloned from `https://github.com/erigontech/erigon.git`
+  into the hive client directory. Example: `/hive-test api branch=fix/my-feature`
+- **eest-version=VERSION** - Pin EEST fixtures version (e.g. `v5.3.0`). Default: auto-discover latest.
+- **bal-version=VERSION** - Pin BAL fixtures version (e.g. `bal@v5.1.0`). Default: auto-discover latest.
+
+## Expected Failures (CI thresholds)
+
+| Suite | Max Allowed Failures |
+|-------|---------------------|
+| exchange-capabilities | 0 |
+| withdrawals | 0 |
+| cancun | 3 |
+| api | 0 |
+| auth | 0 |
+| rpc-compat | 23 |
+| eest (consume-engine) | 0 |
+| eest-bal | 4 (upstream BPO2ToAmsterdamAtTime15k fork transition) |
+
+Note: Failure counts are version-dependent and may change with newer fixtures.
+The eest-bal fork transition failures are a known upstream issue, not Erigon bugs.
+
+## Procedure
+
+### Phase 0: Discover Versions
+
+Before setup, discover the latest EEST fixture versions from GitHub. Skip this phase
+if the user provided explicit version overrides (`eest-version=`, `bal-version=`).
+
+```bash
+# Latest standard EEST fixtures (tag matching v*.*.*)
+EEST_VERSION=$(curl -s https://api.github.com/repos/ethereum/execution-spec-tests/releases \
+  | jq -r '[.[] | select(.tag_name | test("^v[0-9]+\\.[0-9]+\\.[0-9]+$"))][0].tag_name')
+
+# Latest BAL fixtures (tag matching bal@v*.*.*)
+BAL_TAG=$(curl -s https://api.github.com/repos/ethereum/execution-spec-tests/releases \
+  | jq -r '[.[] | select(.tag_name | startswith("bal@"))][0].tag_name')
+BAL_BRANCH="tests-${BAL_TAG}"
+BAL_TAG_URLENC=$(echo "$BAL_TAG" | sed 's/@/%40/')
+BAL_FIXTURES_URL="https://github.com/ethereum/execution-spec-tests/releases/download/${BAL_TAG_URLENC}/fixtures_bal.tar.gz"
+```
+
+Then check whether the EEST mapper at the discovered tag already has the exception
+entries Erigon needs. If not, the `disable_strict_exception_matching` workaround is
+required:
+
+```bash
+MAPPER_URL="https://raw.githubusercontent.com/ethereum/execution-specs/refs/tags/${BAL_BRANCH}/packages/testing/src/execution_testing/client_clis/clis/erigon.py"
+if curl -sf "$MAPPER_URL" | grep -q "GAS_USED_OVERFLOW"; then
+    DISABLE_STRICT=""
+    echo "Mapper has BAL exception entries — strict matching enabled"
+else
+    DISABLE_STRICT='--sim.buildarg disable_strict_exception_matching=erigon'
+    echo "Mapper missing BAL exception entries — strict matching disabled for erigon"
+fi
+```
+
+Log the discovered versions:
+```
+EEST: $EEST_VERSION | BAL: $BAL_TAG (branch: $BAL_BRANCH) | Strict matching: enabled/disabled
+```
+
+### Phase 1: Setup
+
+1. **Determine erigon source path.** Default is the current git working directory.
+   Verify it contains a `Makefile` and `go.mod` with `erigontech/erigon`.
+
+2. **Choose a work directory.** Use `mktemp -d /tmp/hive-test-XXXXXX` for isolation.
+
+3. **Clone hive:**
+   ```bash
+   WORKDIR=$(mktemp -d /tmp/hive-test-XXXXXX)
+   cd "$WORKDIR"
+   git clone --depth 1 https://github.com/ethereum/hive.git
+   cd hive
+   ```
+
+4. **Copy or clone the erigon source into hive:**
+
+   If `branch=BRANCH` was specified, clone that branch:
+   ```bash
+   git clone --depth 1 --branch "$BRANCH" \
+     https://github.com/erigontech/erigon.git clients/erigon/erigon
+   ```
+
+   Otherwise, copy the local source:
+   ```bash
+   # Use rsync to copy, excluding build artifacts and .git
+   rsync -a --exclude='.git' --exclude='build/' --exclude='temp/' \
+     "$ERIGON_PATH/" clients/erigon/erigon/
+   ```
+
+5. **Install Dockerfile.local** for local builds:
+   Ensure `clients/erigon/Dockerfile.local` exists with the correct content.
+   Key requirements:
+   - Base image: `golang:1.25.0-trixie` (Debian, not Alpine)
+   - Build command: `make EXTRA_BUILD_TAGS=nosilkworm erigon`
+   - Runtime: `debian:13-slim` with `bash curl jq libstdc++6 libgcc-s1`
+   - P2P protocol: `erigon.sh` must include `--p2p.protocol 68,69`
+
+   If `clients/erigon/Dockerfile.local` doesn't already exist or doesn't have
+   `nosilkworm`, write the correct version:
+   ```dockerfile
+   FROM golang:1.25.0-trixie as builder
+   ARG local_path=erigon
+   COPY $local_path erigon
+   RUN apt-get update && apt-get install -y bash build-essential ca-certificates git \
+       && cd erigon && make EXTRA_BUILD_TAGS=nosilkworm erigon \
+       && cp build/bin/erigon /usr/local/bin/erigon
+
+   FROM debian:13-slim
+   COPY --from=builder /usr/local/bin/erigon /usr/local/bin/
+   RUN apt-get update && apt-get install -y bash curl jq libstdc++6 libgcc-s1 && rm -rf /var/lib/apt/lists/*
+   RUN erigon --version | sed -e 's/erigon version \(.*\)/\1/' > /version.txt
+   COPY genesis.json /genesis.json
+   COPY mapper.jq /mapper.jq
+   COPY erigon.sh /erigon.sh
+   COPY enode.sh /hive-bin/enode.sh
+   RUN chmod +x /erigon.sh /hive-bin/enode.sh
+   EXPOSE 8545 8546 8551 30303 30303/udp
+   ENTRYPOINT ["/erigon.sh"]
+   ```
+
+6. **P2P protocol configuration:**
+   Do NOT add `--p2p.protocol` flags to erigon.sh. The hive devp2p simulator
+   does not yet support eth/69, so advertising it causes Fork ID test failures
+   (`rlp: expected input list for devp2p.Disconnect`). Let erigon use its default
+   protocol negotiation.
+
+6b. **Parallel execution for Amsterdam (BAL) blocks:**
+   Verify that `erigon.sh` enables parallel execution when Amsterdam is configured.
+   BAL validation only runs in the parallel execution path. If the following block
+   is missing from `erigon.sh`, add it after the `--sync.parallel-state-flushing` line:
+   ```bash
+   # Enable parallel execution for Amsterdam (BAL) blocks.
+   # BAL validation only runs in the parallel execution path.
+   if [ "$HIVE_AMSTERDAM_TIMESTAMP" != "" ]; then
+       export ERIGON_EXEC3_PARALLEL=true
+   fi
+   ```
+   Without this, Amsterdam blocks run through serial execution which has no BAL
+   validation, causing `test_bal_invalid` tests to incorrectly return VALID.
+
+7. **Create client config file:**
+   ```bash
+   cat > erigon-local.yaml <<'EOF'
+   - client: erigon
+     dockerfile: local
+   EOF
+   ```
+
+8. **Build hive binary:**
+   ```bash
+   go build .
+   ```
+
+### Phase 2: Run Tests
+
+Always use `--sim.parallelism 12` for all suites to maximize throughput.
+
+When running multiple suites, launch **separate hive sessions in parallel** (as
+background shell commands) whenever the suites use different simulators. This gives
+`runs × parallelism` total concurrency. Suites using the same simulator can be combined
+with `--sim.limit "suite1|suite2|..."`.
+
+**Engine suites** (sim: `ethereum/engine`) — combine all with `|`:
+```bash
+./hive --client-file erigon-local.yaml --sim ethereum/engine \
+  --sim.limit "exchange-capabilities|withdrawals|cancun|api|auth" \
+  --sim.parallelism 12 --sim.timelimit 30m
+```
+
+**RPC compat** (sim: `ethereum/rpc`, limit: `compat`):
+```bash
+./hive --client-file erigon-local.yaml --sim ethereum/rpc \
+  --sim.limit "compat" --sim.parallelism 12 --sim.timelimit 30m
+```
+
+**EEST** (sim: `ethereum/eels/consume-engine`):
+```bash
+# $EEST_VERSION discovered in Phase 0 (e.g. v5.4.0), or user-provided via eest-version=
+./hive --client-file erigon-local.yaml \
+  --sim ethereum/eels/consume-engine \
+  --sim.parallelism=12 --docker.nocache=true \
+  --sim.buildarg fixtures=https://github.com/ethereum/execution-spec-tests/releases/download/${EEST_VERSION}/fixtures_develop.tar.gz \
+  --sim.timelimit 60m
+```
+
+**EEST BAL** (sim: `ethereum/eels/consume-engine`, amsterdam filter):
+```bash
+# $BAL_BRANCH, $BAL_FIXTURES_URL, $DISABLE_STRICT discovered in Phase 0
+# (e.g. BAL_BRANCH=tests-bal@v5.2.0), or user-provided via bal-version=
+./hive --client-file erigon-local.yaml \
+  --sim ethereum/eels/consume-engine \
+  --sim.limit=".*amsterdam.*" \
+  --sim.parallelism=12 --docker.nocache=true \
+  --sim.buildarg branch=${BAL_BRANCH} \
+  --sim.buildarg fixtures=${BAL_FIXTURES_URL} \
+  ${DISABLE_STRICT} \
+  --sim.timelimit 60m
+```
+
+Note: The `disable_strict_exception_matching` flag is only added when Phase 0 detects
+that the EEST `ErigonExceptionMapper` at the discovered tag is missing required entries
+(e.g. `BlockException.GAS_USED_OVERFLOW`, BAL exception types). Once upstream updates
+their tagged releases with the mapper fixes (already on `forks/amsterdam` HEAD), this
+flag will no longer be needed automatically.
+
+### Phase 3: Parse Results
+
+After each suite, parse the output to extract results:
+```bash
+status_line=$(tail -2 output.log | head -1 | sed -r "s/\x1B\[[0-9;]*[a-zA-Z]//g")
+suites=$(echo "$status_line" | sed -n 's/.*suites=\([0-9]*\).*/\1/p')
+tests=$(echo "$status_line" | sed -n 's/.*tests=\([0-9]*\).*/\1/p')
+failed=$(echo "$status_line" | sed -n 's/.*failed=\([0-9]*\).*/\1/p')
+```
+
+Also check the JSON result files in `workspace/logs/*.json` for detailed per-test
+results.  Report pass/fail counts and list any failing test names.
+
+### Phase 4: Cleanup
+
+**Always run cleanup**, even if tests fail:
+
+```bash
+# Clean up hive containers
+./hive --cleanup
+
+# Optionally remove the work directory
+rm -rf "$WORKDIR"
+
+# Prune dangling docker images from the test run
+docker image prune -f
+```
+
+## Troubleshooting
+
+### Container crash: "libsilkworm_capi.so not found"
+The Dockerfile.local must use `make EXTRA_BUILD_TAGS=nosilkworm erigon`. Ensure the
+Dockerfile does NOT use Alpine (use Debian trixie).
+
+### Fork ID test failures: "rlp: expected input list for devp2p.Disconnect"
+The erigon.sh is missing eth/69 support. Ensure `--p2p.protocol 68,69`.
+
+### Timeout failures
+Run suites separately instead of combining them. Increase `--sim.timelimit` if needed.
+
+### Leftover containers
+Run `./hive --cleanup` or `docker rm -f $(docker ps -aq)` to remove stale containers.
+The hive binary has built-in cleanup: `./hive --cleanup --cleanup.older-than 1h`.
