automatic upgrades

.github/workflows/ci.yml

@@ -44,6 +44,8 @@ jobs:
             core:
               - 'core/**'
               - '!core/CHANGELOG.md'
+              - 'contracts'
+              - 'contracts/**'
               - 'docker/contract-verifier/**'
               - 'docker/external-node/**'
               - 'docker/server/**'

AGENTS.md

@@ -0,0 +1,191 @@
+# ZK Stack Development Guidelines
+
+## Rebuilding After Changes
+
+### When to Rebuild zkstackup
+
+**IMPORTANT:** After making changes to Rust code in the `zkstack_cli` directory, you must rebuild zkstackup for the
+changes to take effect:
+
+```bash
+cd /path/to/zksync-working
+zkstackup --local
+```
+
+This is necessary because:
+
+- The `zkstack` command is a compiled binary installed in `~/.local/bin/`
+- Changes to Rust source code won't take effect until the binary is rebuilt
+- Without rebuilding, you'll be running the old version of the CLI
+
+**When to rebuild:**
+
+- After modifying any `.rs` files in `zkstack_cli/`
+- After modifying Forge script parameters in `zkstack_cli/crates/config/src/forge_interface/script_params.rs`
+- After making changes to upgrade command handlers
+
+## Solidity Upgrade Scripts
+
+### Ecosystem Upgrade Architecture
+
+The v31 ecosystem upgrade uses a unified approach that combines both core contract upgrades and CTM (Chain Type Manager)
+upgrades:
+
+```
+EcosystemUpgrade_v31
+    ↓ extends
+DefaultEcosystemUpgrade
+    ↓ extends                    ↓ has instance of
+DefaultCoreUpgrade          CTMUpgrade_v31
+                                ↓ extends
+                            DefaultCTMUpgrade
+```
+
+**Key Features:**
+
+- `DefaultEcosystemUpgrade` runs both core and CTM upgrades sequentially
+- Combines governance calls from both upgrades into unified stage0/1/2 calls
+- Copies diamond cut data from CTM upgrade to ecosystem output file
+- Avoids diamond inheritance conflicts by using composition (CTM upgrade as state variable)
+
+**Files:**
+
+- `deploy-scripts/upgrade/default_upgrade/DefaultEcosystemUpgrade.s.sol` - Base class for unified upgrades
+- `deploy-scripts/upgrade/v31/EcosystemUpgrade_v31.s.sol` - v31-specific implementation
+- `deploy-scripts/upgrade/v31/CTMUpgrade_v31.s.sol` - v31 CTM upgrade
+- `deploy-scripts/upgrade/v31/CoreUpgrade_v31.s.sol` - Standalone core upgrade (not used by ecosystem upgrade)
+
+**Environment Variables:**
+
+- `V31_UPGRADE_ECOSYSTEM_OUTPUT` - Main output file path (e.g., `/script-out/v31-upgrade-core.toml`)
+- `V31_UPGRADE_CTM_OUTPUT` - CTM output file path (e.g., `/script-out/v31-upgrade-ctm.toml`)
+
+**Output Files:**
+
+- The ecosystem output file (`v31-upgrade-core.toml`) contains:
+  - Ecosystem contract addresses
+  - CTM contract addresses
+  - **Diamond cut data** for chain upgrades
+  - Combined governance calls for all upgrade stages
+
+## Debugging Forge Scripts
+
+### Common Issues
+
+1. **"call to non-contract address 0x0"**
+
+   - Usually means a contract hasn't been deployed or registered yet
+   - Check if required contracts exist at the expected addresses
+   - For upgrades: ensure chains are registered before running upgrade scripts
+
+2. **"vm.writeToml: path not allowed"**
+
+   - Check that paths are correctly constructed (relative vs absolute)
+   - Ensure `vm.projectRoot()` is only concatenated once
+   - Environment variable paths like `/script-out/...` are relative to project root
+
+3. **Missing diamond cut data**
+   - Ensure both core and CTM upgrades are running
+   - Verify `saveCombinedOutput()` is called after CTM upgrade completes
+   - Check that CTM output file is being read correctly
+
+### Debugging Failed Transactions with `cast run`
+
+When you encounter "missing revert data" or unclear transaction failures, use this method to get the full execution
+trace:
+
+**Step 1: Extract transaction details from error**
+
+From an error like:
+
+```
+transaction={ "data": "0xd52471c1...", "from": "0x97D2A9...", "to": "0xfe3EE966..." }
+```
+
+**Step 2: Send the transaction manually with sufficient gas**
+
+```bash
+TX_HASH=$(cast send <TO_ADDRESS> \
+  "<CALLDATA>" \
+  --value <VALUE_IF_NEEDED> \
+  --private-key <PRIVATE_KEY> \
+  --rpc-url http://127.0.0.1:8545 \
+  --gas-price 50gwei \
+  --gas-limit 10000000 2>&1 | grep "transactionHash" | awk '{print $2}')
+```
+
+**Important**: Use `--gas-limit 10000000` to ensure the transaction gets mined even if it reverts. This allows us to
+trace it.
+
+**Step 3: Trace the transaction to see where it failed**
+
+```bash
+cast run $TX_HASH --rpc-url http://127.0.0.1:8545
+```
+
+This will show the full call trace with:
+
+- Every contract call in the execution path
+- Function names and parameters
+- Where exactly the revert occurred
+- The revert reason (e.g., "call to non-contract address 0x0000...")
+
+**Example**:
+
+```bash
+# From error message, extract: to=0xfe3EE966..., data=0xd52471c1..., value=1050000121535147500000
+
+TX_HASH=$(cast send 0xfe3EE966E7790b427F7B078f304C7B4DDCd4bbfe \
+  "0xd52471c10000000000000000000000000000000000000000000000000000000000000020..." \
+  --value 1050000121535147500000 \
+  --private-key 0x7726827caac94a7f9e1b160f7ea819f172f7b6f9d2a97f992c38edeab82d4110 \
+  --rpc-url http://127.0.0.1:8545 \
+  --gas-price 50gwei \
+  --gas-limit 10000000 2>&1 | grep "transactionHash" | awk '{print $2}')
+
+cast run $TX_HASH --rpc-url http://127.0.0.1:8545
+```
+
+This will output:
+
+```
+Traces:
+  [99306] Bridgehub::requestL2TransactionDirect(...)
+    ├─ [92138] BridgehubImpl::requestL2TransactionDirect(...) [delegatecall]
+    │   ├─ [70552] L1AssetRouter::bridgehubDepositBaseToken(...)
+    │   │   ├─ [63432] L1AssetRouterImpl::bridgehubDepositBaseToken(...) [delegatecall]
+    │   │   │   ├─ [48684] NativeTokenVault::bridgeBurn(...)
+    │   │   │   │   └─ ← [Revert] call to non-contract address 0x0000000000000000000000000000000000000000
+```
+
+**Key benefits**:
+
+- Shows the exact call path leading to the failure
+- Reveals which contract call failed and why
+- Makes it clear if a required contract is missing or uninitialized
+- Much more informative than "missing revert data" errors
+
+## General Rules
+
+### NEVER USE try-catch OR staticcall in Upgrade Scripts
+
+**THIS IS AN ABSOLUTE RULE - NO EXCEPTIONS**
+
+❌ **FORBIDDEN PATTERNS:**
+
+- `try contract.someFunction() { ... } catch { ... }`
+- `(bool ok, bytes memory data) = target.staticcall(...)`
+
+✅ **CORRECT APPROACH:**
+
+- If a function reverts, fix the root cause (missing deployment, wrong order, etc.)
+- Check if contracts exist before calling them: `if (address != address(0)) { ... }`
+- Query protocol version or initialization state
+- Restructure when the script runs
+
+**WHY THIS RULE EXISTS:**
+
+- try-catch and staticcall hide real errors instead of fixing them
+- These patterns make debugging extremely difficult
+- They mask initialization issues and timing problems
+- The codebase should fail fast and clearly, not silently return defaults

CLAUDE.md

@@ -0,0 +1 @@
+@AGENTS.md

core/lib/dal/src/blocks_dal.rs

@@ -24,6 +24,7 @@ use zksync_types::{
     },
     commitment::{L1BatchCommitmentArtifacts, L1BatchWithMetadata, PubdataParams},
     l2_to_l1_log::{BatchAndChainMerklePath, UserL2ToL1Log},
+    settlement::SettlementLayer,
     writes::TreeWrite,
     Address, Bloom, L1BatchNumber, L2BlockNumber, ProtocolVersionId, SLChainId, H256, U256,
 };
@@ -241,6 +242,16 @@ impl BlocksDal<'_, '_> {
         Ok(Some(header.into()))
     }
 
+    /// Returns latest sealed L1 batch header. Returns `None` if there are no sealed batches.
+    pub async fn get_latest_sealed_l1_batch_header(
+        &mut self,
+    ) -> DalResult<Option<CommonL1BatchHeader>> {
+        let Some(number) = self.get_sealed_l1_batch_number().await? else {
+            return Ok(None);
+        };
+        self.get_common_l1_batch_header(number).await
+    }
+
     pub async fn get_sealed_l2_block_number(&mut self) -> DalResult<Option<L2BlockNumber>> {
         let row = sqlx::query!(
             r#"
@@ -3207,6 +3218,47 @@ impl BlocksDal<'_, '_> {
         Ok(count != 0)
     }
 
+    /// Returns `true` if there is any batch on the provided settlement layer that isn't fully committed yet.
+    /// This includes unsealed batches and sealed batches without a finalized commit tx.
+    pub async fn has_uncommitted_batches_on_settlement_layer(
+        &mut self,
+        settlement_layer: &SettlementLayer,
+    ) -> DalResult<bool> {
+        let (settlement_layer_type, settlement_layer_chain_id) =
+            from_settlement_layer(settlement_layer);
+        let count = sqlx::query_scalar!(
+            r#"
+            SELECT COUNT(*)
+            FROM l1_batches
+            WHERE
+                settlement_layer_type = $1
+                AND settlement_layer_chain_id = $2
+                AND (
+                    NOT is_sealed
+                    OR l1_batches.eth_commit_tx_id IS NULL
+                    OR NOT EXISTS (
+                        SELECT 1
+                        FROM eth_txs_history
+                        WHERE
+                            eth_tx_id = l1_batches.eth_commit_tx_id
+                            AND sent_successfully = TRUE
+                            AND finality_status = 'finalized'
+                    )
+                )
+            "#,
+            settlement_layer_type.as_str(),
+            settlement_layer_chain_id
+        )
+        .instrument("has_uncommitted_batches_on_settlement_layer")
+        .with_arg("settlement_layer_type", &settlement_layer_type)
+        .with_arg("settlement_layer_chain_id", &settlement_layer_chain_id)
+        .fetch_one(self.storage)
+        .await?
+        .unwrap_or(0);
+
+        Ok(count != 0)
+    }
+
     // methods used for measuring Eth tx stage transition latencies
     // and emitting metrics base on these measured data
     pub async fn oldest_uncommitted_batch_timestamp(&mut self) -> DalResult<Option<u64>> {

core/node/api_server/src/web3/namespaces/unstable/mod.rs

@@ -8,14 +8,12 @@ use zksync_dal::{Connection, Core, CoreDal, DalError};
 use zksync_mini_merkle_tree::MiniMerkleTree;
 use zksync_multivm::{interface::VmEvent, zk_evm_latest::ethereum_types::U64};
 use zksync_types::{
-    aggregated_operations::L1BatchAggregatedActionType,
     api,
     api::{
         ChainAggProof, DataAvailabilityDetails, GatewayMigrationStatus, L1ToL2TxsStatus, TeeProof,
         TransactionDetailedResult, TransactionExecutionInfo,
     },
-    eth_sender::EthTxFinalityStatus,
-    server_notification::GatewayMigrationState,
+    server_notification::{GatewayMigrationNotification, GatewayMigrationState},
     tee_types::TeeType,
     web3,
     web3::Bytes,
@@ -250,34 +248,41 @@ impl UnstableNamespace {
             .await
             .map_err(DalError::generalize)?;
 
-        let all_batches_with_interop_roots_committed = match connection
-            .interop_root_dal()
-            .get_latest_processed_interop_root_l1_batch_number()
-            .await
-            .map_err(DalError::generalize)?
-        {
-            None => true,
-            Some(latest_processed_l1_batch_number) => {
-                match connection
-                    .eth_sender_dal()
-                    .get_last_sent_successfully_eth_tx_by_batch_and_op(
-                        L1BatchNumber::from(latest_processed_l1_batch_number),
-                        L1BatchAggregatedActionType::Commit,
-                    )
-                    .await
-                {
-                    Some(tx) => tx.eth_tx_finality_status == EthTxFinalityStatus::Finalized,
-                    None => false,
-                }
-            }
-        };
         let state = GatewayMigrationState::from_sl_and_notification(
             self.state
                 .api_config
                 .settlement_layer
                 .settlement_layer_for_sending_txs(),
             latest_notification,
         );
+        let settlement_layer = self.state.api_config.settlement_layer.settlement_layer();
+        let has_uncommitted_batches = connection
+            .blocks_dal()
+            .has_uncommitted_batches_on_settlement_layer(&settlement_layer)
+            .await
+            .map_err(DalError::generalize)?;
+        let latest_sealed_matches_expected = match (
+            latest_notification,
+            connection
+                .blocks_dal()
+                .get_latest_sealed_l1_batch_header()
+                .await
+                .map_err(DalError::generalize)?,
+        ) {
+            (Some(GatewayMigrationNotification::ToGateway), Some(header)) => {
+                header.settlement_layer.is_gateway()
+            }
+            (Some(GatewayMigrationNotification::FromGateway), Some(header)) => {
+                !header.settlement_layer.is_gateway()
+            }
+            (Some(_), None) => false,
+            (None, _) => true,
+        };
+        let all_batches_committed = if state == GatewayMigrationState::InProgress {
+            !has_uncommitted_batches
+        } else {
+            !has_uncommitted_batches && latest_sealed_matches_expected
+        };
 
         Ok(GatewayMigrationStatus {
             latest_notification,
@@ -287,7 +292,7 @@ impl UnstableNamespace {
                 .api_config
                 .settlement_layer
                 .settlement_layer_for_sending_txs(),
-            wait_for_batches_to_be_committed: !all_batches_with_interop_roots_committed,
+            wait_for_batches_to_be_committed: !all_batches_committed,
         })
     }