update docs

lispc · lispc · commit c424e53cf4fb · 2026-06-03T11:56:03.000+08:00
diff --git a/AGENTS.md b/AGENTS.md
@@ -25,9 +25,9 @@ Follow the structured testing guide in [`docs/testing/openvm-upgrade-testing-gui
 For testing proof generation against **real mainnet production tasks** without interfering with the live system, use the **Shadow Coordinator** approach. This is significantly faster than a full shadow fork:
 
 - **Architecture**: Local coordinator (`:8390`) + local prover (GPU), fed by imported production task data.
-- **Docs**: [`docs/shadow-testing/README.md`](docs/shadow-testing/README.md) — full setup guide, troubleshooting, config reference.
-- **Quick Start**: [`scripts/shadow-testing/QUICKSTART.md`](scripts/shadow-testing/QUICKSTART.md)
-- **Automation**: [`scripts/shadow-testing/setup.sh`](scripts/shadow-testing/setup.sh) — one-command setup for postgres, coordinator, and prover.
+- **Docs**: [`tests/shadow-testing/docs/README.md`](tests/shadow-testing/docs/README.md) — full setup guide, troubleshooting, config reference.
+- **Quick Start**: [`tests/shadow-testing/docs/QUICKSTART.md`](tests/shadow-testing/docs/QUICKSTART.md)
+- **Automation**: [`tests/shadow-testing/scripts/setup.sh`](tests/shadow-testing/scripts/setup.sh) — one-command setup for postgres, coordinator, and prover.
 
 Key hard-won rules:
 - **L2 RPC for coordinator task generation** (must support `debug_executionWitness`):
@@ -153,5 +153,5 @@ make coordinator_setup
 | [`docs/prover-coordinator-overview.md`](docs/prover-coordinator-overview.md) | Architecture, data flow, component relationships, common operations |
 | [`docs/testing/openvm-upgrade-testing-guide.md`](docs/testing/openvm-upgrade-testing-guide.md) | Step-by-step testing checklist after OpenVM / zkvm-prover upgrades |
 | [`docs/testing/docker-compose-e2e-guide.md`](docs/testing/docker-compose-e2e-guide.md) | Production-like E2E testing with Docker Compose + Coordinator Proxy |
-| [`docs/shadow-testing/README.md`](docs/shadow-testing/README.md) | Shadow coordinator + local prover setup for production task replay |
+| [`tests/shadow-testing/docs/README.md`](tests/shadow-testing/docs/README.md) | Shadow coordinator + local prover setup for production task replay |
 | [`docs/testing_reports/openvm-v1.6.0-guest-v0.8.0-May19.md`](docs/testing_reports/openvm-v1.6.0-guest-v0.8.0-May19.md) | Test report for PR #1783 (OpenVM 1.6.0, guest v0.8.0) |
diff --git a/crates/libzkp/src/proofs.rs b/crates/libzkp/src/proofs.rs
@@ -339,3 +339,4 @@ mod tests {
         Ok(())
     }
 }
+
diff --git a/rollup/internal/controller/sender/estimategas.go b/rollup/internal/controller/sender/estimategas.go
@@ -102,6 +102,11 @@ func (s *Sender) estimateBlobGas(to *common.Address, data []byte, sidecar *types
 }
 
 func (s *Sender) estimateGasLimit(to *common.Address, data []byte, sidecar *types.BlobTxSidecar, gasPrice, gasTipCap, gasFeeCap, blobGasFeeCap *big.Int) (uint64, *types.AccessList, error) {
+	// In dry-run mode, skip gas estimation and use a fixed gas limit.
+	if s.config.DryRun {
+		return 10000000, nil, nil
+	}
+
 	msg := ethereum.CallMsg{
 		From:      s.transactionSigner.GetAddr(),
 		To:        to,
diff --git a/scripts/shadow-testing/QUICKSTART.md b/scripts/shadow-testing/QUICKSTART.md
@@ -1,84 +1,3 @@
 # Quick Start: Shadow Coordinator + Prover
 
-For full details, see `docs/shadow-testing/README.md`.
-
-## Prerequisites
-
-1. **IDC port-forward active**: Mainnet RDS on `localhost:15432`
-2. **Docker installed** with GPU support (for prover)
-3. **Verifier assets** at `/tmp/shadow-verifier-assets/` (feynman, galileo, galileoV2)
-4. **SRS params** at `~/.openvm/params/` (kzg_bn254_22.srs, kzg_bn254_23.srs, kzg_bn254_24.srs)
-
-## One-Command Setup
-
-```bash
-cd scripts/shadow-testing
-
-# Step 1: Start PostgreSQL
-./setup.sh --postgres
-
-# Step 2: Import production tasks (requires RDS port-forward)
-./import-production-data.sh
-
-# Step 3: Fetch L2 block headers
-python3 fetch-l2-blocks.py \
-  --rpc https://mainnet-rpc.scroll.io \
-  --db "postgresql://<user>:<password>@localhost:5433/shadow_rollup" \
-  --start-block 33750000 --end-block 33770000
-
-# Step 4: Link blocks to chunks
-psql "postgresql://<user>:<password>@localhost:5433/shadow_rollup" -c "
-  UPDATE l2_block lb SET chunk_hash = c.hash
-  FROM chunk c
-  WHERE lb.number >= c.start_block_number AND lb.number <= c.end_block_number;
-"
-
-# Step 5: Start coordinator (takes 2-3 min)
-./setup.sh --coordinator
-
-# Step 6: Start prover (in another terminal)
-./setup.sh --prover
-```
-
-## Monitoring
-
-```bash
-# Check everything is running
-./setup.sh --status
-
-# Watch coordinator logs
-docker logs -f shadow-coordinator-api-test --tail 100
-
-# Watch prover logs (if using docker)
-docker logs -f shadow-prover --tail 100
-
-# Check task assignment
-psql "postgresql://<user>:<password>@localhost:5433/shadow_rollup" -c "
-  SELECT proving_status, COUNT(*) FROM chunk GROUP BY proving_status;
-"
-```
-
-## Stop Everything
-
-```bash
-./setup.sh --stop
-```
-
-## Key Configuration Files
-
-| File | Purpose |
-|------|---------|
-| `configs/shadow-coordinator-config.json` | Coordinator config template |
-| `configs/prover-local.json` | Prover config template |
-| `/tmp/shadow-coordinator-config.json` | Generated coordinator config (with L2 RPC) |
-| `/tmp/prover-local.json` | Generated prover config |
-
-## Environment Variables
-
-| Variable | Default | Description |
-|----------|---------|-------------|
-| `PROD_DB` | `postgresql://...localhost:15432/rollup` | Production RDS connection |
-| `SHADOW_DB` | `postgresql://...localhost:5433/shadow_rollup` | Shadow DB connection |
-| `VERIFIER_DIR` | `/tmp/shadow-verifier-assets` | Verifier asset path |
-| `IMAGE_TAG` | `v4.7.13-openvm16` | Docker image tag |
-| `L2_RPC` | `https://mainnet-rpc.scroll.io` | L2 RPC endpoint |
+> **Moved**: This document has been moved to [`tests/shadow-testing/docs/QUICKSTART.md`](../../tests/shadow-testing/docs/QUICKSTART.md).
diff --git a/tests/shadow-testing/README.md b/tests/shadow-testing/README.md
@@ -384,19 +384,70 @@ The single biggest time sink was using the **wrong selector** for `finalizeBundl
 cast sig "finalizeBundlePostEuclidV2(bytes,uint256,bytes32,bytes32,bytes)"
 ```
 
-### Trap #2: Verifier Digest Mismatch
+### Trap #2: Verifier Mismatch — Digests *and* Plonk Verifier
 
-Mainnet's registered verifier (`0x0dE180…`) has digests `0x009160…` / `0x009305…`, but the proofs generated by our local prover (guest v0.8.0) require digests `0x00398b…` / `0x002178…`.
+This is the most subtle trap. There are **two** independent verifier components:
 
-**Symptom**: Direct verifier call returns `VerificationFailed()`.
+1. **`ZkEvmVerifierPostEuclid` / `ZkEvmVerifierPostFeynman`** — the Solidity wrapper that assembles public inputs and calls the plonk verifier.
+2. **The plonk verifier contract** — a ~18KB (new guest) or ~37KB (mainnet legacy) EVM bytecode that performs the actual SNARK verification.
+
+**Mainnet's registered verifier** (`0x0dE180…`) has:
+- Digests: `0x009160…` / `0x009305…`
+- Plonk verifier: `0x749fC77A…` (37KB legacy bytecode)
+
+**Our locally-generated proofs** (guest v0.8.0) require:
+- Digests: extracted from the proof's `instances` array (see below)
+- Plonk verifier: `coordinator/build/bin/assets_v2/verifier.bin` (18KB new bytecode)
+
+**Symptom**: `VerificationFailed(0x439cc0cd)` even when digests *appear* correct. This happens because the plonk verifier bytecode itself differs between mainnet (37KB) and the new guest release (18KB). The wrapper may pass the right digests, but the underlying plonk verifier rejects the proof format.
+
+**How to determine the correct verifier for your proofs**
+
+The proof's `instances` array tells you exactly which wrapper + plonk verifier you need:
+
+```python
+import base64, json
+
+# proof_json from coordinator/bundle table
+instances = base64.b64decode(proof_json['proof']['instances'])
+print(f"instances length: {len(instances)} bytes")  # e.g. 1472
+
+# For ZkEvmVerifierPostEuclid (new guest v0.8.0+):
+#   1472 bytes = 12 accumulators (384) + 2 digests (64) + 32 publicInputHash bytes (1024)
+#   = 46 × 32-byte Fr elements
+digest1 = '0x' + instances[384:416].hex()
+digest2 = '0x' + instances[416:448].hex()
+```
+
+If `len(instances) == 1472` (or more generally `12+2+32 = 46` words), your proof is for **`ZkEvmVerifierPostEuclid`**. The wrapper computes `keccak256(publicInput)` and feeds each of the 32 hash bytes as a separate field element to the plonk verifier.
+
+If `len(instances)` were smaller (e.g. 12+13 = 25 words), the proof would be for `ZkEvmVerifierPostFeynman`, which decomposes public inputs into 13 field elements directly.
+
+**Fix — Automated deployment**
+
+A script handles all three steps (deploy plonk, deploy wrapper, register):
 
-**Fix**: Deploy a new `ZkEvmVerifierPostEuclid` (not `PostFeynman`!) with matching digests:
 ```bash
-# In scroll-contracts/
-cast send <DEPLOYER> "createZkEvmVerifierPostEuclid(address,bytes32,bytes32)" \
-  <PLONK_VERIFIER> <DIGEST1> <DIGEST2> --rpc-url <ANVIL_RPC>
+# From tests/shadow-testing/
+./scripts/02-deploy-verifier.sh --config configs/mainnet.json
 ```
-Then register it via `MultipleVersionRollupVerifier.updateVerifier(10, <START_BATCH>, <NEW_VERIFIER>)`.
+
+What it does:
+1. Reads `coordinator/build/bin/assets_v2/verifier.bin` and deploys it as the new plonk verifier via `cast send --create`.
+2. Queries the shadow DB for the bundle proof, base64-decodes `instances`, and extracts `digest1`/`digest2` from offsets `[384:416]` and `[416:448]`.
+3. Compiles and deploys `ZkEvmVerifierPostFeynman` (not `PostEuclid`!) with `forge create` using the new plonk address + extracted digests + `protocolVersion = 10`.
+4. Calls `MultipleVersionRollupVerifier.updateVerifier(10, startBatch, newWrapper)`.
+5. Updates `configs/mainnet.json` → `contracts.deployed_verifier`.
+
+**Why `ZkEvmVerifierPostFeynman` and not `PostEuclid`**
+
+`PostFeynman` computes `keccak256(abi.encodePacked(protocolVersion, publicInput))`, where `protocolVersion = (domain << 6) + stf_version` = `(0 << 6) + 10 = 10` for Scroll domain + V10 STF version. This matches exactly how the prover computes `bundle_pi_hash` in `pi_hash_versioned()`.
+
+`PostEuclid` computes `keccak256(publicInput)` *without* the `protocolVersion` prefix, so the hash never matches the proof's embedded `bundle_pi_hash`. The 32-byte `protocolVersion` padding is visible in the proof's `instances` array at offset `[448:1472]` (32 words, each containing one byte of the hash).
+
+**Why you cannot reuse the mainnet plonk verifier**
+
+Mainnet's plonk verifier (`0x749fC77A…`, 37KB) was compiled for the old Halo2 circuit. The new guest v0.8.0 uses a different Halo2 outer circuit with a smaller verification key. The 18KB `verifier.bin` from `assets_v2/` is the *only* plonk verifier that understands the new proof format. Copying mainnet bytecode via `anvil_setCode` only works if you are verifying **mainnet-generated proofs**; it will always fail for **locally-generated proofs** from a newer guest.
 
 ### Trap #3: `committedBatches` Is Sparse (EuclidV2)
 
diff --git a/tests/shadow-testing/docs/LESSONS_LEARNED.md b/tests/shadow-testing/docs/LESSONS_LEARNED.md
@@ -0,0 +1,37 @@
+# Shadow Testing Lessons Learned
+
+## 2026-06-03: Production data import overwrote locally-valid shadow proofs
+
+### What Happened
+1.  Bundles 17302-17304 had already been successfully committed and finalized on the shadow fork using proofs generated by the local shadow code.
+2.  After discovering batch-hash mismatches, we re-imported bundles 17302:17339 from production RDS.
+3.  **Mistake**: The import script did **not** exclude the `proof` column, so production proofs (generated with the production circuit version) overwrote the locally-valid shadow proofs.
+4.  The production proof verifier digests (`0x0091609a...` / `0x009305f0...`) did **not** match the shadow verifier digests (`0x00398b78...` / `0x0021785a...`).
+5.  Result: Bundle 17305's finalize transaction reverted with `VerificationFailed()` (custom error `0x439cc0cd`).
+
+### Root Cause
+- The production and shadow test environments were running different zkvm-prover / circuit versions.
+- Proofs are tightly coupled to circuit versions: each proof embeds verifier digests that must exactly match the verifier contract deployed on-chain.
+- The data import blindly copied entire tables without protecting the shadow-local proofs.
+
+### How to Prevent This
+1. **Always exclude `proof` columns when importing production data**. Correct approach:
+   - Import raw data (headers, transactions, state roots, etc.) for batches, chunks, and L2 blocks.
+   - **Do NOT import `bundle.proof`, `batch.proof`, or `chunk.proof`.**
+   - After import, reset `proving_status` to `ProvingTaskUnassigned` so the coordinator re-schedules proving tasks.
+
+2. **Back up shadow-local valid proofs before importing** (if you need to keep them).
+
+3. **Verify digest compatibility before finalizing**: Extract the digests from the proof and compare them against the on-chain verifier contract's `verifierDigest1()` / `verifierDigest2()`. Only send the finalize transaction after confirming they match.
+
+### Recovery Steps
+1.  Clear the `proof` field for bundles 17305+.
+2.  Reset `proving_status` and `batch_proofs_status` to pending.
+3.  Reset proving status for the corresponding batches and chunks as well.
+4.  Restart the coordinator and prover so they regenerate proofs matching the shadow verifier.
+
+### Pre-Import Checklist
+- [ ] Confirm the import script excludes `proof` columns.
+- [ ] Confirm `proving_status` is reset to `ProvingTaskUnassigned` for bundles/batches/chunks after import.
+- [ ] Confirm shadow verifier digests match the expected circuit version.
+- [ ] Confirm prover and coordinator can communicate (JWT tokens are valid).
diff --git a/tests/shadow-testing/docs/QUICKSTART.md b/tests/shadow-testing/docs/QUICKSTART.md
@@ -0,0 +1,84 @@
+# Quick Start: Shadow Coordinator + Prover
+
+For full details, see `tests/shadow-testing/docs/README.md`.
+
+## Prerequisites
+
+1. **IDC port-forward active**: Mainnet RDS on `localhost:15432`
+2. **Docker installed** with GPU support (for prover)
+3. **Verifier assets** at `/tmp/shadow-verifier-assets/` (feynman, galileo, galileoV2)
+4. **SRS params** at `~/.openvm/params/` (kzg_bn254_22.srs, kzg_bn254_23.srs, kzg_bn254_24.srs)
+
+## One-Command Setup
+
+```bash
+cd tests/shadow-testing
+
+# Step 1: Start PostgreSQL
+./scripts/setup.sh --postgres
+
+# Step 2: Import production tasks (requires RDS port-forward)
+./scripts/import-production-data.sh
+
+# Step 3: Fetch L2 block headers
+python3 ./scripts/fetch-l2-blocks.py \
+  --rpc https://mainnet-rpc.scroll.io \
+  --db "postgresql://<user>:<password>@localhost:5433/shadow_rollup" \
+  --start-block 33750000 --end-block 33770000
+
+# Step 4: Link blocks to chunks
+psql "postgresql://<user>:<password>@localhost:5433/shadow_rollup" -c "
+  UPDATE l2_block lb SET chunk_hash = c.hash
+  FROM chunk c
+  WHERE lb.number >= c.start_block_number AND lb.number <= c.end_block_number;
+"
+
+# Step 5: Start coordinator (takes 2-3 min)
+./scripts/setup.sh --coordinator
+
+# Step 6: Start prover (in another terminal)
+./scripts/setup.sh --prover
+```
+
+## Monitoring
+
+```bash
+# Check everything is running
+./scripts/setup.sh --status
+
+# Watch coordinator logs
+docker logs -f shadow-coordinator-api-test --tail 100
+
+# Watch prover logs (if using docker)
+docker logs -f shadow-prover --tail 100
+
+# Check task assignment
+psql "postgresql://<user>:<password>@localhost:5433/shadow_rollup" -c "
+  SELECT proving_status, COUNT(*) FROM chunk GROUP BY proving_status;
+"
+```
+
+## Stop Everything
+
+```bash
+./scripts/setup.sh --stop
+```
+
+## Key Configuration Files
+
+| File | Purpose |
+|------|---------|
+| `configs/shadow-coordinator-config.json` | Coordinator config template |
+| `configs/prover-local.json` | Prover config template |
+| `/tmp/shadow-coordinator-config.json` | Generated coordinator config (with L2 RPC) |
+| `/tmp/prover-local.json` | Generated prover config |
+
+## Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `PROD_DB` | `postgresql://...localhost:15432/rollup` | Production RDS connection |
+| `SHADOW_DB` | `postgresql://...localhost:5433/shadow_rollup` | Shadow DB connection |
+| `VERIFIER_DIR` | `/tmp/shadow-verifier-assets` | Verifier asset path |
+| `IMAGE_TAG` | `v4.7.13-openvm16` | Docker image tag |
+| `L2_RPC` | `https://mainnet-rpc.scroll.io` | L2 RPC endpoint |
diff --git a/tests/shadow-testing/docs/README.md b/tests/shadow-testing/docs/README.md
@@ -43,7 +43,7 @@ If you just want to get running, use the provided script:
 
 ```bash
 # 1. Set up shadow PostgreSQL
-cd scripts/shadow-testing
+cd tests/shadow-testing
 ./setup.sh --postgres
 
 # 2. Import production task data (requires RDS port-forward)
@@ -120,7 +120,7 @@ Export the latest N batches + their chunks + bundles from production RDS and imp
 
 ```bash
 # Edit these variables as needed
-# Credentials loaded from .env (see scripts/shadow-testing/.env.example)
+# Credentials loaded from .env (see tests/shadow-testing/.env.example)
 PROD_DB="postgresql://${PROD_DB_USER}:${PROD_DB_PASSWORD}@${PROD_DB_HOST}:${PROD_DB_PORT}/${PROD_DB_NAME}"
 SHADOW_DB="postgresql://${SHADOW_DB_USER}:${SHADOW_DB_PASSWORD}@${SHADOW_DB_HOST}:${SHADOW_DB_PORT}/${SHADOW_DB_NAME}"
 BATCH_LIMIT=50
@@ -174,7 +174,7 @@ The coordinator needs `l2_block` records to format chunk tasks (for block hashes
 Use the provided Python script or fetch blocks via L2 RPC:
 
 ```bash
-python3 scripts/shadow-testing/fetch-l2-blocks.py \
+python3 tests/shadow-testing/scripts/fetch-l2-blocks.py \
   --rpc https://mainnet-rpc.scroll.io \
   --db "postgresql://$SHADOW_DB_USER:$SHADOW_DB_PASSWORD@$SHADOW_DB_HOST:$SHADOW_DB_PORT/$SHADOW_DB_NAME" \
   --start-block 26000000 \
@@ -396,7 +396,7 @@ When `"dry_run": true` is set in the sender config:
 cd rollup && go build -o rollup_relayer ./cmd/rollup_relayer/app
 ```
 
-2. Configure `dry_run: true` in the sender config (see `scripts/shadow-testing/configs/rollup-relayer-dryrun.json`)
+2. Configure `dry_run: true` in the sender config (see `tests/shadow-testing/configs/rollup-relayer-dryrun.json`)
 
 3. Start the relayer:
 ```bash
diff --git a/tests/shadow-testing/docs/contract-addresses.md b/tests/shadow-testing/docs/contract-addresses.md

Original file line number	Diff line number	Diff line change
`@@ -339,3 +339,4 @@ mod tests {`
`339`	`339`	`Ok(())`
`340`	`340`	`}`
`341`	`341`	`}`
	`342`	`+`