feat(permissionless batches): batch production toolkit and operator recovery #1555
+2,271
−31
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,3 +4,5 @@ docs/ | |
| l2geth/ | ||
| rpc-gateway/ | ||
| *target/* | ||
|
|
||
| permissionless-batches/conf/ | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,3 +4,5 @@ docs/ | |
| l2geth/ | ||
| rpc-gateway/ | ||
| *target/* | ||
|
|
||
| permissionless-batches/conf/ | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,3 +4,5 @@ docs/ | |
| l2geth/ | ||
| rpc-gateway/ | ||
| *target/* | ||
|
|
||
| permissionless-batches/conf/ | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,8 @@ | ||
| assets/ | ||
| contracts/ | ||
| docs/ | ||
| l2geth/ | ||
| rpc-gateway/ | ||
| *target/* | ||
|
|
||
| permissionless-batches/conf/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,30 @@ | ||
| # Download Go dependencies | ||
| FROM scrolltech/go-rust-builder:go-1.21-rust-nightly-2023-12-03 as base | ||
|
|
||
| WORKDIR /src | ||
| COPY go.work* ./ | ||
| COPY ./rollup/go.* ./rollup/ | ||
| COPY ./common/go.* ./common/ | ||
| COPY ./coordinator/go.* ./coordinator/ | ||
| COPY ./database/go.* ./database/ | ||
| COPY ./tests/integration-test/go.* ./tests/integration-test/ | ||
| COPY ./bridge-history-api/go.* ./bridge-history-api/ | ||
| RUN go mod download -x | ||
|
|
||
| # Build rollup_relayer | ||
| FROM base as builder | ||
|
|
||
| RUN --mount=target=. \ | ||
| --mount=type=cache,target=/root/.cache/go-build \ | ||
| cd /src/rollup/cmd/permissionless_batches/ && CGO_LDFLAGS="-ldl" go build -v -p 4 -o /bin/rollup_relayer | ||
|
|
||
| # Pull rollup_relayer into a second stage deploy ubuntu container | ||
| FROM ubuntu:20.04 | ||
|
|
||
| RUN apt update && apt install vim netcat-openbsd net-tools curl ca-certificates -y | ||
|
|
||
| ENV CGO_LDFLAGS="-ldl" | ||
|
|
||
| COPY --from=builder /bin/rollup_relayer /bin/ | ||
| WORKDIR /app | ||
| ENTRYPOINT ["rollup_relayer"] |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| assets/ | ||
| contracts/ | ||
| docs/ | ||
| l2geth/ | ||
| rpc-gateway/ | ||
| *target/* | ||
|
|
||
| permissionless-batches/conf/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,8 @@ | ||
| assets/ | ||
| contracts/ | ||
| docs/ | ||
| l2geth/ | ||
| rpc-gateway/ | ||
| *target/* | ||
|
|
||
| permissionless-batches/conf/ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| conf/ | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,172 @@ | ||
| # Permissionless Batches | ||
| Permissionless batches aka enforced batches is a feature that provides guarantee to users that they can exit Scroll even if the operator is down or censoring. | ||
| It allows anyone to take over and submit a batch (permissionless batch submission) together with a proof after a certain time period has passed without a batch being finalized on L1. | ||
|
|
||
| Once permissionless batch mode is activated, the operator can no longer submit batches in a permissioned way. Only the security council can deactivate permissionless batch mode and reinstate the operator as the only batch submitter. | ||
| There are two types of situations to consider: | ||
| - `Permissionless batch mode is activated:` This means that finalization halted for some time. Now anyone can submit batches utilizing the [batch production toolkit](#batch-production-toolkit). | ||
| - `Permissionless batch mode is deactivated:` This means that the security council has decided to reinstate the operator as the only batch submitter. The operator needs to [recover](#operator-recovery) the sequencer and relayer to resume batch submission and the valid L2 chain. | ||
|
|
||
|
|
||
| ## Batch production toolkit | ||
| The batch production toolkit is a set of tools that allow anyone to submit a batch in permissionless mode. It consists of three main components: | ||
| 1. l2geth state recovery from L1 | ||
| 2. l2geth block production | ||
| 3. production, proving and submission of batch with `docker-compose.yml` | ||
|
|
||
| ### Prerequisites | ||
| - Unix-like OS, 32GB RAM | ||
| - Docker | ||
| - [l2geth](https://github.com/scroll-tech/go-ethereum/) or [Docker image](https://hub.docker.com/r/scrolltech/l2geth) of corresponding version [TODO link list with versions](#batch-production-toolkit). | ||
| - access to an Ethereum L1 RPC node (beacon node and execution client) | ||
| - ability to run a prover or access to a proving service (e.g. Sindri) | ||
| - L1 account with funds to pay for the batch submission | ||
|
|
||
| ### 1. l2geth state recovery from L1 | ||
| Once permissionless mode is activated there's no blocks being produced and propagated on L2. The first step is to recover the latest state of the L2 chain from L1. This is done by running l2geth in recovery mode. | ||
| More information about l2geth recovery (aka L1 follower mode) can be found [here TODO: put correct link once released](https://github.com/scroll-tech/scroll-documentation/pull/374). | ||
|
|
||
| Running l2geth in recovery mode requires following configuration: | ||
| - `--scroll` or `--scroll-sepolia` - enables Scroll Mainnet or Sepolia mode | ||
| - `--da.blob.beaconnode` - L1 RPC beacon node | ||
| - `--l1.endpoint` - L1 RPC execution client | ||
| - `--da.sync=true` - enables syncing with L1 | ||
| - `--da.recovery` - enables recovery mode | ||
| - `--da.recovery.initiall1block` - initial L1 block (commit tx of initial batch) | ||
| - `--da.recovery.initialbatch` - batch where to start recovery from. Can be found on [Scrollscan Explorer](https://scrollscan.com/batches). | ||
| - `--da.recovery.l2endblock` - until which L2 block recovery should run (optional) | ||
|
|
||
| ```bash | ||
| ./build/bin/geth --scroll<-sepolia> \ | ||
| --datadir "tmp/datadir" \ | ||
| --gcmode archive \ | ||
| --http --http.addr "0.0.0.0" --http.port 8545 --http.api "eth,net,web3,debug,scroll" --http.vhosts "*" \ | ||
| --da.blob.beaconnode "<L1 RPC beacon node>" \ | ||
| --l1.endpoint "<L1 RPC execution client>" \ | ||
| --da.sync=true --da.recovery --da.recovery.initiall1block "<initial L1 block (commit tx of initial batch)>" --da.recovery.initialbatch "<batch where to start recovery from>" --da.recovery.l2endblock "<until which L2 block recovery should run (optional)>" \ | ||
| --verbosity 3 | ||
| ``` | ||
|
|
||
| ### 2. l2geth block production | ||
| After the state is recovered, the next step is to produce blocks on L2. This is done by running l2geth in block production mode. | ||
| As a prerequisite, the state recovery must be completed and the latest state of the L2 chain must be available. | ||
|
|
||
| You also need to generate a keystore e.g. with [Clef](https://geth.ethereum.org/docs/fundamentals/account-management) to be able to sign blocks. | ||
| This key is not used for any funds, but required for block production to work. Once you generated blocks you can safely discard it. | ||
|
|
||
| Running l2geth in block production mode requires following configuration: | ||
| - `--scroll` or `--scroll-sepolia` - enables Scroll Mainnet or Sepolia mode | ||
| - `--da.blob.beaconnode` - L1 RPC beacon node | ||
| - `--l1.endpoint` - L1 RPC execution client | ||
| - `--da.sync=true` - enables syncing with L1 | ||
| - `--da.recovery` - enables recovery mode | ||
| - `--da.recovery.produceblocks` - enables block production | ||
| - `--miner.etherbase '0xeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee' --mine` - enables mining. the address is not used, but required for mining to work | ||
| - `---miner.gaslimit 1 --miner.gasprice 1 --miner.maxaccountsnum 100 --rpc.gascap 0 --gpo.ignoreprice 1` - gas limits for block production | ||
|
|
||
| ```bash | ||
| ./build/bin/geth --scroll<-sepolia> \ | ||
| --datadir "tmp/datadir" \ | ||
| --gcmode archive \ | ||
| --http --http.addr "0.0.0.0" --http.port 8545 --http.api "eth,net,web3,debug,scroll" --http.vhosts "*" \ | ||
| --da.blob.beaconnode "<L1 RPC beacon node>" \ | ||
| --l1.endpoint "<L1 RPC execution client>" \ | ||
| --da.sync=true --da.recovery --da.recovery.produceblocks \ | ||
| --miner.gaslimit 1 --miner.gasprice 1 --miner.maxaccountsnum 100 --rpc.gascap 0 --gpo.ignoreprice 1 \ | ||
| --miner.etherbase '0xeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee' --mine \ | ||
| --ccc \ | ||
| --verbosity 3 | ||
|
Comment on lines
+65
to
+78
There was a problem hiding this comment. Stray The sample command includes |
||
| ``` | ||
|
|
||
| ### 3. production, proving and submission of batch with `docker-compose.yml` | ||
| After the blocks are produced, the next step is to produce a batch, prove it and submit it to L1. This is done by running the `docker-compose.yml` in the `permissionless-batches` folder. | ||
|
|
||
|
|
||
| #### Producing a batch | ||
| To produce a batch you need to run the `batch-production-submission` profile in `docker-compose.yml`. | ||
|
|
||
| 1. Fill `conf/genesis.json` with the latest genesis state from the L2 chain. The genesis for the current fork can be found here: [TODO link list with versions](#batch-production-toolkit) | ||
| 2. Make sure that `l2geth` with your locally produced blocks is running and reachable from the Docker network (e.g. `http://host.docker.internal:8545`) | ||
| 3. Fill in required fields in `conf/relayer/config.json` | ||
|
|
||
|
|
||
| Run with `docker compose --profile batch-production-submission up`. | ||
| This will produce chunks, a batch and bundle which will be proven in the next step. | ||
| `Success! You're ready to generate proofs!` indicates that everything is working correctly and the batch is ready to be proven. | ||
|
|
||
| #### Proving a batch | ||
| To prove the chunk, batch and bundle you just generated you need to run the `proving` profile in `docker-compose.yml`. | ||
|
|
||
| 1. Make sure `verifier` `low_version_circuit` and `high_version_circuit` in `conf/coordinator/config.json` are correct for the latest fork: [TODO link list with versions](#batch-production-toolkit) | ||
| 2. Download the latest `assets` and `params` for the circuit from [TODO link list with versions](#batch-production-toolkit) into `conf/coordinator/assets` and `conf/coordinator/params` respectively. | ||
| 3. Fill in the required fields in `conf/proving-service/config.json`. It is recommended to use Sindri. You'll need to obtain credits and an API key from their [website](https://sindri.app/). | ||
| 4. Alternatively, you can run your own prover: https://github.com/scroll-tech/scroll-prover. However, this requires more configuration. | ||
|
|
||
| Run with `docker compose --profile proving up`. | ||
|
|
||
|
|
||
| #### Batch submission | ||
| To submit the batch you need to run the `batch-production-submission` profile in `docker-compose.yml`. | ||
|
|
||
| 1. Fill in required fields in `conf/relayer/config.json` for the sender config. | ||
|
|
||
| Run with `docker compose --profile batch-production-submission up`. | ||
| This will submit the batch to L1 and finalize it. The transaction will be retried in case of failure. | ||
|
|
||
| **Troubleshooting** | ||
| - in case the submission fails it will print the calldata for the transaction in an error message. You can use this with `cast call --trace --rpc-url "$SCROLL_L1_DEPLOYMENT_RPC" "$L1_SCROLL_CHAIN_PROXY_ADDR" <calldata>` to see what went wrong. | ||
| - `0x4df567b9: ErrorNotInEnforcedBatchMode`: permissionless batch mode is not activated, you can't submit a batch | ||
| - `0xa5d305cc: ErrorBatchIsEmpty`: no blob was provided. This is usually returned if you do the `cast call`, permissionless mode is activated but you didn't provide a blob in the transaction. | ||
|
|
||
| ## Operator recovery | ||
| Operator recovery needs to be run by the rollup operator to resume normal rollup operation after permissionless batch mode is deactivated. It consists of two main components: | ||
| 1. l2geth recovery | ||
| 2. Relayer recovery | ||
|
|
||
| These steps are required to resume permissioned batch submission and the valid L2 chain. They will restore the entire history of the batches submitted during permissionless mode. | ||
|
|
||
| ### Prerequisites | ||
| - l2geth with the latest state of the L2 chain (before permissionless mode was activated) | ||
| - signer key for the sequencer according to Clique consensus | ||
| - relayer and coordinator are set up, running and up-to-date with the latest state of the L2 chain (before permissionless mode was activated) | ||
|
|
||
| ### l2geth recovery | ||
| Running l2geth in recovery mode requires following configuration: | ||
| - `--scroll` or `--scroll-sepolia` - enables Scroll Mainnet or Sepolia mode | ||
| - `--da.blob.beaconnode` - L1 RPC beacon node | ||
| - `--l1.endpoint` - L1 RPC execution client | ||
| - `--da.sync=true` - enables syncing with L1 | ||
| - `--da.recovery` - enables recovery mode | ||
| - `--da.recovery.signblocks` - enables signing blocks with the sequencer and configured key | ||
| - `--da.recovery.initiall1block` - initial L1 block (commit tx of initial batch) | ||
| - `--da.recovery.initialbatch` - batch where to start recovery from. Can be found on [Scrollscan Explorer](https://scrollscan.com/batches). | ||
| - `--da.recovery.l2endblock` - until which L2 block recovery should run (optional) | ||
|
|
||
| ```bash | ||
| ./build/bin/geth --scroll<-sepolia> \ | ||
| --datadir "tmp/datadir" \ | ||
| --gcmode archive \ | ||
| --http --http.addr "0.0.0.0" --http.port 8545 --http.api "eth,net,web3,debug,scroll" --http.vhosts "*" \ | ||
| --da.blob.beaconnode "<L1 RPC beacon node>" \ | ||
| --l1.endpoint "<L1 RPC execution client>" \ | ||
| --da.sync=true --da.recovery --da.recovery.signblocks --da.recovery.initiall1block "<initial L1 block (commit tx of initial batch)>" --da.recovery.initialbatch "<batch where to start recovery from>" --da.recovery.l2endblock "<until which L2 block recovery should run (optional)>" \ | ||
| --verbosity 3 | ||
| ``` | ||
|
|
||
| After the recovery is finished, start the sequencer in normal operation and continue issuing L2 blocks as normal. This will resume the L2 chain, allow the relayer (after running recovery) to create new batches and allow other L2 follower nodes to sync up the valid and signed L2 chain. | ||
|
|
||
| ### Relayer recovery | ||
| Start the relayer with the following additional top-level configuration: | ||
| ``` | ||
| "recovery_config": { | ||
| "enable": true | ||
| } | ||
| ``` | ||
|
|
||
| This will make the relayer recover all the chunks, batches and bundles that were submitted during permissionless mode. These batches are marked automatically as proven and finalized. | ||
| Once this process is finished, start the relayer normally without the recovery config to resume normal operation. | ||
| ``` | ||
| "recovery_config": { | ||
| "enable": false | ||
| } | ||
| ``` | ||
Empty file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,38 @@ | ||||||
| { | ||||||
| "prover_manager": { | ||||||
| "provers_per_session": 1, | ||||||
| "session_attempts": 5, | ||||||
| "bundle_collection_time_sec": 3600, | ||||||
| "batch_collection_time_sec": 3600, | ||||||
| "chunk_collection_time_sec": 3600, | ||||||
| "verifier": { | ||||||
| "mock_mode": false, | ||||||
| "low_version_circuit": { | ||||||
| "params_path": "./conf/params", | ||||||
| "assets_path": "./conf/assets", | ||||||
| "fork_name": "darwinV2", | ||||||
| "min_prover_version": "v4.4.55" | ||||||
| }, | ||||||
| "high_version_circuit": { | ||||||
| "params_path": "./conf/params", | ||||||
| "assets_path": "./conf/assets", | ||||||
| "fork_name": "darwinV2", | ||||||
| "min_prover_version": "v4.4.56" | ||||||
| } | ||||||
| } | ||||||
| }, | ||||||
| "db": { | ||||||
| "driver_name": "postgres", | ||||||
| "dsn": "postgres://db/scroll?sslmode=disable&user=postgres", | ||||||
yiweichi marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
| "maxOpenNum": 200, | ||||||
| "maxIdleNum": 20 | ||||||
| }, | ||||||
| "l2": { | ||||||
| "chain_id": 111 | ||||||
| }, | ||||||
| "auth": { | ||||||
| "secret": "prover secret key", | ||||||
|
There was a problem hiding this comment. 🛠️ Refactor suggestion Authentication secret should not be hardcoded. The authentication secret "prover secret key" appears to be a placeholder and should be replaced with a secure, randomly generated value in production. Replace the hardcoded secret with an environment variable reference: - "secret": "prover secret key",
+ "secret": "${AUTH_SECRET}",Then ensure this secret is provided through a secure method like environment variables or a secrets management system. 📝 Committable suggestion
Suggested change
|
||||||
| "challenge_expire_duration_sec": 3600, | ||||||
| "login_expire_duration_sec": 3600 | ||||||
| } | ||||||
| } | ||||||
Empty file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| <fill with correct genesis.json> | ||
yiweichi marked this conversation as resolved.
Show resolved
Hide resolved
|
||
Empty file.
Empty file.
Empty file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| { | ||
| "prover_name_prefix": "prover_", | ||
| "keys_dir": "/app/", | ||
| "db_path": "/app/", | ||
| "coordinator": { | ||
| "base_url": "http://coordinator:8390", | ||
| "retry_count": 3, | ||
| "retry_wait_time_sec": 5, | ||
| "connection_timeout_sec": 60 | ||
| }, | ||
| "l2geth": { | ||
| "endpoint": "<L2 RPC with generated blocks reachable from Docker network>" | ||
yiweichi marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| }, | ||
| "prover": { | ||
| "circuit_type": 2, | ||
| "circuit_version": "v0.13.1", | ||
| "n_workers": 1, | ||
| "cloud": { | ||
| "base_url": "https://sindri.app/api/v1/", | ||
| "api_key": "<API key>", | ||
yiweichi marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| "retry_count": 3, | ||
| "retry_wait_time_sec": 5, | ||
| "connection_timeout_sec": 60 | ||
| } | ||
| } | ||
| } | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.