lambdaclass · ilitteri · Aug 24, 2026 · Aug 24, 2026 · Aug 24, 2026 · Aug 24, 2026
@@ -5,6 +5,7 @@ This is a per-release acceptance test. The goal is to verify that:
 1. A node running the **previous release** (`<VERSION_FROM>`) can be cleanly stopped.
 2. The L1 contracts can be upgraded following the per-release migration guide.
 3. The **new release** (`<VERSION_TO>`) sequencer and prover resume operation against the upgraded contracts without re-deploying from scratch.
+4. Batches produced by `<VERSION_FROM>` and committed by `<VERSION_TO>` are **proved and verified on L1** under the new release's verification key.
 
 The commands below are ready to copy-paste. The only thing you need to substitute is the version tags. Every other value (private keys, addresses, ports, fee parameters) is taken from the defaults in `crates/l2/Makefile`; feel free to override them, but the values below give you a working baseline.
 
@@ -15,13 +16,27 @@ Replace these once at the top of your shell session:
 ```bash
 export VERSION_FROM=<previous release tag, e.g. v11.0.0>
 export VERSION_TO=<new release tag, e.g. v12.0.0-rc.2>
-export ARCH=linux-x86_64       # or linux-aarch64, macos-aarch64
+export ARCH=linux-x86_64
 export WORK=$HOME/upgrade-test  # workspace
 mkdir -p "$WORK"
 ```
 
+> [!IMPORTANT]
+> Run this on the GPU server (`l2-gpu`) with the **SP1** prover, not locally with
+> `--backend exec`. The `exec` backend produces no proof, so it never consults a
+> verification key — and the verification key is the thing an upgrade changes.
+> Batches are committed under `keccak(VERGEN_GIT_SHA)`, baked into the binary at
+> build time, and `commitBatch` refuses a commit hash it holds no key for. Until
+> `$VERSION_TO`'s key is registered
+> ([Step 3.6](#36-register-the-version_to-verification-key)) the upgraded
+> sequencer cannot commit **anything**: every `commitBatch` reverts with
+> `MissingVerificationKeyForCommit()` (selector `0xf6b9798e`) and the L2 keeps
+> producing blocks locally while nothing reaches L1. Under `exec` that failure is
+> invisible — the test passes and the real upgrade stops at the first batch.
+
 ## Prerequisites
 
+- A CUDA-capable GPU (this is why it runs on `l2-gpu`), and the `ethrex-l2-$ARCH-gpu` release asset
 - [`rex`](https://github.com/lambdaclass/rex) on `$PATH` (used to call `upgradeToAndCall` and read contract state)
 - `curl`, `jq`
 - `git` (to fetch genesis / fixture files for each version)
@@ -71,16 +86,26 @@ cd "$WORK"
 git clone --branch "$VERSION_FROM" --depth 1 https://github.com/lambdaclass/ethrex.git "ethrex-$VERSION_FROM"
 git clone --branch "$VERSION_TO"   --depth 1 https://github.com/lambdaclass/ethrex.git "ethrex-$VERSION_TO"
 
-# Binaries
-curl -L "https://github.com/lambdaclass/ethrex/releases/download/$VERSION_FROM/ethrex-l2-$ARCH" -o "ethrex-$VERSION_FROM/ethrex"
-curl -L "https://github.com/lambdaclass/ethrex/releases/download/$VERSION_TO/ethrex-l2-$ARCH"   -o "ethrex-$VERSION_TO/ethrex"
-chmod +x "ethrex-$VERSION_FROM/ethrex" "ethrex-$VERSION_TO/ethrex"
+# Binaries. The `-gpu` asset is the one that can prove with SP1 on a GPU.
+# Remove before downloading: curl-ing over a binary that is currently running
+# reuses the inode and gets the live process killed.
+for V in "$VERSION_FROM" "$VERSION_TO"; do
+  rm -f "ethrex-$V/ethrex"
+  curl -fL "https://github.com/lambdaclass/ethrex/releases/download/$V/ethrex-l2-$ARCH-gpu" -o "ethrex-$V/ethrex"
+  chmod +x "ethrex-$V/ethrex"
+  "ethrex-$V/ethrex" --version
+done
 
-"ethrex-$VERSION_FROM/ethrex" --version
-"ethrex-$VERSION_TO/ethrex"   --version
+# Verification keys ship in each release's contracts tarball; both versions'
+# keys are needed — VERSION_FROM's to deploy, VERSION_TO's for Step 3.6.
+for V in "$VERSION_FROM" "$VERSION_TO"; do
+  mkdir -p "contracts-$V"
+  curl -fL "https://github.com/lambdaclass/ethrex/releases/download/$V/ethrex-contracts.tar.gz" \
+    | tar xz -C "contracts-$V"
+done
 ```
 
-> Use the `ethrex-l2-*` asset (the plain `ethrex-*` asset is an L1-only build and does not have the `l2` subcommand).
+> Use an `ethrex-l2-*` asset (the plain `ethrex-*` asset is an L1-only build and does not have the `l2` subcommand), and specifically the `-gpu` variant — the others carry no CUDA support, so `--backend sp1` falls back to CPU proving and a batch takes long enough to look like a hang.
 
 ### 0.1 Save the `$VERSION_FROM` L2 genesis
 
@@ -125,13 +150,21 @@ COMPILE_CONTRACTS=true ./ethrex l2 deploy \
   --on-chain-proposer-owner 0x4417092b70a3e5f10dc504d0947dd256b965fc62 \
   --bridge-owner 0x4417092b70a3e5f10dc504d0947dd256b965fc62 \
   --bridge-owner-pk 0x941e103320615d394a55708be13e45994c7d93b932b064dbcb2b511fe3254e2e \
+  --sp1 true \
+  --sp1-vk-path "$WORK/contracts-$VERSION_FROM/ethrex-riscv32im-succinct-zkvm-vk-bn254" \
   --deposit-rich \
   --private-keys-file-path fixtures/keys/private_keys_l1.txt \
   --genesis-l1-path fixtures/genesis/l1.json \
   --genesis-l2-path fixtures/genesis/l2.json \
   --env-file-path cmd/.env
 ```
 
+`--sp1 true` is what makes the OnChainProposer require a real proof, and
+`--sp1-vk-path` registers `$VERSION_FROM`'s verification key against
+`$VERSION_FROM`'s commit hash. Both matter: without them the deployment accepts
+anything and [Step 3.6](#36-register-the-version_to-verification-key) has nothing
+to prove.
+
 After this finishes, capture the addresses we'll need later:
 
 ```bash
@@ -176,7 +209,7 @@ set -a; source cmd/.env; set +a
 cd "$WORK/ethrex-$VERSION_FROM"
 ./ethrex l2 prover \
   --proof-coordinators tcp://127.0.0.1:3900 \
-  --backend exec
+  --backend sp1
 ```
 
 ### 1.5 Confirm the stack is healthy
@@ -405,6 +438,60 @@ rex send "$ETHREX_WATCHER_BRIDGE_ADDRESS" 'setL2GasLimit(uint256)' <NEW_GAS_LIMI
   --rpc-url http://localhost:8545
 ```
 
+### 3.6 Register the `$VERSION_TO` verification key
+
+Required on every release bump, not just the ones that change contracts.
+
+Each batch carries `keccak(VERGEN_GIT_SHA)` of the binary that committed it, and
+`commitBatch` rejects a commit hash with no registered key before it stores
+anything. The `$VERSION_FROM` deployer registered only `$VERSION_FROM`'s key, so
+until this step runs the upgraded sequencer commits nothing at all — the
+committer logs
+
+```
+Failed to send commitment for batch N ... execution reverted: 0xf6b9798e
+```
+
+which is `MissingVerificationKeyForCommit()`. Both `lastCommittedBatch` and
+`lastVerifiedBatch` sit still while the L2 keeps producing blocks, so the symptom
+looks like a stuck committer rather than a key problem.
+
+Both arguments come from the release you are upgrading to:
+
+```bash
+# The hashed value is the ASCII sha the binary reports, not the tag.
+TO_SHA=$("$WORK/ethrex-$VERSION_TO/ethrex" --version | sed 's|.*HEAD-||; s|/.*||')
+TO_COMMIT_HASH=$(cast keccak "$TO_SHA")
+
+# The vk is the same hex-text file the deployer takes via --sp1-vk-path,
+# from $VERSION_TO's ethrex-contracts.tar.gz.
+TO_VK=0x$(tr -d '\n' < ethrex-riscv32im-succinct-zkvm-vk-bn254 | sed 's/^0x//')
+```
+
+`rex l2 register-vk` derives the commit hash, routes the call through the
+Timelock, and reads the key back afterwards:
+
+```bash
+rex l2 register-vk \
+  --commit "$TO_SHA" \
+  --vk "$TO_VK" \
+  --on-chain-proposer "$ETHREX_COMMITTER_ON_CHAIN_PROPOSER_ADDRESS" \
+  --timelock "$ETHREX_TIMELOCK_ADDRESS" \
+  --private-key 0x941e103320615d394a55708be13e45994c7d93b932b064dbcb2b511fe3254e2e \
+  --rpc-url http://localhost:8545
+```
+
+The `--timelock` flag is what makes this work: `upgradeSP1VerificationKey` is
+`onlyOwner`, that owner is the Timelock, and the command routes through
+`emergencyExecute` — which needs no delay because in this test the Security
+Council is the account passed as `--on-chain-proposer-owner`. Sending straight to
+the OnChainProposer instead reverts with `OwnableUnauthorizedAccount`. See
+[Timelock](../../l2/fundamentals/timelock.md) for the Governance
+`schedule` + `execute` route.
+
+Run it with `--dry-run` first if you want to see the derived commit hash and the
+key currently on chain without sending anything.
+
 ---
 
 ## Step 4 — Run the `$VERSION_TO` stack
@@ -447,7 +534,7 @@ Note:
 
 ```bash
 cd "$WORK/ethrex-$VERSION_TO"
-./ethrex l2 prover --proof-coordinators tcp://127.0.0.1:3900 --backend exec
+./ethrex l2 prover --proof-coordinators tcp://127.0.0.1:3900 --backend sp1
 ```
 
 ---

@@ -96,7 +96,7 @@ Before publishing the release, run through the following checks using the pre-re
 - [ ] Upgrade `ethrex-teku`
 - [ ] Upgrade `ethrex-grandine`
 - [ ] Launch multisync on `ethrex-multisync-main`
-- [ ] Upgrade a local L2 created with the previous version and run the integration tests
+- [ ] Upgrade an L2 created with the previous version and run the integration tests, on the GPU server (`l2-gpu`) with the SP1 prover
 - [ ] Run the L2 integration tests with a SP1 prover on the GPU server (`l2-gpu`)
 
 The commands for each target follow. The host roster changes between releases — fill in the ones you run and leave the placeholders for the rest. Replace `vX.Y.Z-rc.W` / `release/vX.Y.Z` with the version under test.
@@ -155,14 +155,29 @@ tmux new-session -d -s sync "make multisync-loop-auto MULTISYNC_BRANCH=release/v
 
 Check progress later with `tmux attach -t sync` (detach with `Ctrl-b` then `d`).
 
-#### Local L2 upgrade + integration tests
+#### L2 upgrade + integration tests (`l2-gpu`)
 
 See [Upgrade test](l2/upgrade-test.md) for the full procedure.
 
+Run it on `l2-gpu` with `--backend sp1`, not locally with `--backend exec`. The
+`exec` backend produces no proof, so it never consults a verification key — and
+the verification key is exactly what an upgrade changes. Batches are committed
+under `keccak(VERGEN_GIT_SHA)`, baked into the binary at build time, and
+`commitBatch` rejects a commit hash the deployment holds no key for, so an
+upgraded sequencer commits nothing at all until the new release's key is
+registered. Under `exec` that whole class of upgrade failure is invisible.
+
+Both L2 checks share the ports and datadirs on `l2-gpu`, so run them one after
+the other, not concurrently.
+
 #### L2 integration tests with a SP1 prover (`l2-gpu`)
 
 See [L2 integration tests with a SP1 GPU prover](l2/sp1-gpu-integration-test.md) for the full procedure.
 
+This one deploys the new version from scratch; the upgrade test above covers the
+path from the previous release. Keep both: a release can deploy cleanly and still
+fail to upgrade, and the reverse.
+
 ### Publish
 
 Once the pre-release is created and you want to publish the release, go to the [release page](https://github.com/lambdaclass/ethrex/releases) and follow the next steps:

@@ -1,5 +1,38 @@
 # Upgrades
 
+## Every release — register the new verification key
+
+Applies to any deployment that verifies real proofs (`--sp1 true` at deploy
+time), whether or not the release changes a contract.
+
+A batch is committed under the git sha of the binary that committed it, and
+`commitBatch` rejects a commit hash the deployment holds no key for. Your
+deployment only holds the key of the version that deployed it, so an upgraded
+sequencer commits **nothing** until the new key is registered — every commit
+reverts with `MissingVerificationKeyForCommit()` (`0xf6b9798e`), surfacing as:
+
+```
+Failed to send commitment for batch N ... execution reverted: 0xf6b9798e
+```
+
+The L2 keeps producing blocks throughout, so it reads as a stuck committer
+rather than a missing key. Nothing is lost: register the key and the pending
+batch commits on the next attempt.
+
+```
+rex l2 register-vk --commit <GIT_SHA> --vk <VERIFICATION_KEY> \
+  --on-chain-proposer <OCP> --timelock <TIMELOCK> --private-key <SECURITY_COUNCIL_PK>
+```
+
+The sha is the one the new binary prints after `HEAD-` in `ethrex --version`; the
+key is the `ethrex-riscv32im-succinct-zkvm-vk-bn254` file from the new release's
+`ethrex-contracts.tar.gz`. See
+[Registering a new verification key](../fundamentals/upgrades.md#registering-a-new-verification-key)
+for what the command does and why the Timelock is in the path.
+
+Keys are stored per commit hash, not per version, so the old key stays valid —
+which is what lets a rollback keep verifying.
+
 ## From v7 to v8
 
 ### Database migration (local node)

@@ -6,25 +6,47 @@ Each committed batch stores the git commit hash of the sequencer build that prod
 
 ## Registering a new verification key
 
-To allow proofs from a new sequencer/prover build, register its verification key against the commit hash:
-
-1. Compute the commit hash as the Keccak-256 of the (reduced) git commit. For example, the commit `9219410` produces `b9105485bc4ba523201eaaf76478a47b259fa7399bbed795cf19294861b7fc57`.
-2. From the OnChainProposer owner account, send the upgrade transaction. Example (replace addresses and keys with your values):
-   ```
-   rex send <ON_CHAIN_PROPOSER_ADDRESS> \
-     "upgradeSP1VerificationKey(bytes32,bytes32)" \
-     <KECCAK_GIT_COMMIT> \
-     <VERIFICATION_KEY> \
-     --private-key <ON_CHAIN_PROPOSER_OWNER_PK>
-   ```
-3. (Optional) Verify the mapping entry:
-   ```
-   rex call <ON_CHAIN_PROPOSER_ADDRESS> \
-     "verificationKeys(bytes32,uint8)(bytes32)" \
-     <KECCAK_GIT_COMMIT> \
-     <PROVER_ID>
-   ```
-   `1` is the SP1 verifier ID, `2` is RISC0.
+This is not optional on an upgrade. `commitBatch` rejects a commit hash it holds
+no key for, so an upgraded sequencer commits **nothing** until its key is
+registered: every commit reverts with `MissingVerificationKeyForCommit()`
+(selector `0xf6b9798e`) while the L2 keeps producing blocks, which reads as a
+stuck committer rather than a missing key. Nothing is lost — register the key and
+the pending batch is committed on the next attempt.
+
+Use `rex l2 register-vk`, which derives the commit hash, routes the call, and
+reads the mapping back:
+
+```
+rex l2 register-vk \
+  --commit <GIT_SHA> \
+  --vk <VERIFICATION_KEY> \
+  --on-chain-proposer <ON_CHAIN_PROPOSER_ADDRESS> \
+  --timelock <TIMELOCK_ADDRESS> \
+  --private-key <SECURITY_COUNCIL_PK>
+```
+
+Add `--prover risc0` for a RISC0 deployment, and `--dry-run` to print the derived
+commit hash and the key currently on chain without sending.
+
+Two details this gets right, and which are easy to get wrong by hand:
+
+- **The hashed value is the full git sha the binary reports**, as ASCII — the
+  segment after `HEAD-` in `ethrex --version`. Hashing an abbreviated sha
+  produces a different key, and the only symptom is that commits keep reverting.
+- **`upgradeSP1VerificationKey` is `onlyOwner`, and that owner is the Timelock**
+  in any deployment made since v9.0.0, so the call must be routed through
+  `emergencyExecute` (Security Council) or Governance `schedule` + `execute`.
+  Sending it straight to the OnChainProposer from an EOA reverts with
+  `OwnableUnauthorizedAccount`.
+
+To read a mapping entry directly (`1` is the SP1 verifier ID, `2` is RISC0):
+
+```
+rex l2 call <ON_CHAIN_PROPOSER_ADDRESS> \
+  "verificationKeys(bytes32,uint8)(bytes32)" \
+  <KECCAK_GIT_COMMIT> \
+  <PROVER_ID>
+```
 
 ### Verification key artifacts