feat(ci): fast integration tests framework #4255

tomg10 · 2025-06-28T21:00:23Z

No description provided.

core/tests/highlevel-test-tools/src/execute-command.ts

core/tests/highlevel-test-tools/src/run-integration-tests.ts

core/tests/highlevel-test-tools/src/start-server.ts

core/tests/highlevel-test-tools/src/run-integration-tests.ts

core/tests/highlevel-test-tools/src/start-external-node.ts

core/tests/highlevel-test-tools/src/run-integration-tests.ts

core/tests/highlevel-test-tools/src/start-external-node.ts

core/tests/highlevel-test-tools/src/start-server.ts

core/tests/highlevel-test-tools/src/execute-command.ts

core/tests/highlevel-test-tools/src/run-integration-tests.ts

core/tests/highlevel-test-tools/src/start-server.ts

core/tests/highlevel-test-tools/src/gateway.ts

core/tests/highlevel-test-tools/src/start-external-node.ts

core/tests/highlevel-test-tools/src/execute-command.ts

+        const logEntry = `[${new Date().toISOString()}] Executing: ${command} ${args.join(' ')}\n`;
+        logStream.write(logEntry);
+
+        const child = spawn(command, args, {


core/tests/highlevel-test-tools/src/gateway.ts

+        try {
+            await executeCommand(
+                'zkstack',
+                ['chain', 'gateway', 'migrate-to-gateway', '--chain', chainName, '--gateway-chain-name', 'gateway'],


core/tests/highlevel-test-tools/src/run-integration-tests.ts

+    try {
+        await executeCommand(
+            'zkstack',
+            ['dev', 'init-test-wallet', '--chain', chainName],


core/tests/highlevel-test-tools/src/run-integration-tests.ts

+        `${emoji} Running ${testName} tests for chain: ${chainName}${testPattern ? ` with pattern: ${testPattern}` : ''}`
+    );
+    const command = 'zkstack';
+    const args = ['dev', 'test', testName, '--no-deps', '--chain', chainName, ...additionalArgs];


core/tests/highlevel-test-tools/src/run-integration-tests.ts

+    const command = 'zkstack';
+    const args = ['dev', 'test', testName, '--no-deps', '--chain', chainName, ...additionalArgs];
+    if (testPattern) {
+        args.push(`--test-pattern='${testPattern}'`);


core/tests/highlevel-test-tools/src/start-external-node.ts

+                `--db-name=${chainName}_external_node`,
+                '--l1-rpc-url=http://localhost:8545',
+                '--chain',
+                chainName,


core/tests/highlevel-test-tools/src/start-external-node.ts

+            // Execute the init command
+            await executeCommand(
+                'zkstack',
+                ['external-node', 'init', '--ignore-prerequisites', '--chain', chainName],


core/tests/highlevel-test-tools/src/start-external-node.ts

+        // Step 1: Run external node with chain-specific arguments
+        console.log(`⏳ Starting external node: ${chainName}`);
+
+        const runArgs = ['external-node', 'run', '--ignore-prerequisites', '--chain', chainName];


core/tests/highlevel-test-tools/src/start-external-node.ts

+        console.log(`⏳ Waiting for external node to be ready: ${chainName}`);
+        await executeCommand(
+            'zkstack',
+            ['external-node', 'wait', '--ignore-prerequisites', '--verbose', '--chain', chainName],


core/tests/highlevel-test-tools/src/start-server.ts

+        // Use executeCommand to wait for the server to be ready
+        await executeCommand(
+            'zkstack',
+            ['server', 'wait', '--ignore-prerequisites', '--verbose', '--chain', chainName],


popzxc

I absolutely see value in the work that has been done here, and it's cool that it brings significant CI speed improvements. I do not object merging this PR.

That said, I find the need in this PR to be a symptom of significant issues with the repo maintenance on many levels, including CI, tooling, and testing. While it does bring improvements, it does so at cost of more (undocumented) complexity, which will also likely suffer from the lack of maintenance, and bring more issues as repo gets more changes.

I would strongly recommend the team to evaluate the level of support they want to provide, and potentially invest into rethinking our approach to CI and testing in general.

core/tests/highlevel-test-tools/package.json

bin/build_and_init_ecosystem

core/tests/highlevel-test-tools/global-setup.ts

Deniallugo

TBH, i don't feel it's the right move at all.

Some commands, like genearte load or wait for batches are good, and probably it's a good command to zkstack dev test.

So, i'd agree to move the additional tools you introduced to zkstack directly.

But other than that, looks like it's an additional complexity with a little value.

we could achieve the same behaviour with the current tooling and with less code

core/tests/highlevel-test-tools/.gitignore

zkstack_cli/crates/zkstack/src/commands/ecosystem/args/init.rs

bin/build_and_init_ecosystem

core/tests/highlevel-test-tools/.gitignore

🤖 I have created a release *beep* *boop* --- ## [28.8.0](core-v28.7.0...core-v28.8.0) (2025-07-07) ### Features * **ci:** fast integration tests framework ([#4255](#4255)) ([a72cbd8](a72cbd8)) * **eigenda:** EigenDA V2 M0 ([#3983](#3983)) ([8302a81](8302a81)) * EN commit, prove, execute batch transactions verification and finality status ([#4080](#4080)) ([d06697d](d06697d)) * **en:** remove JSON RPC syncing ([#4258](#4258)) ([d194604](d194604)) * **state-keeper:** implement block commit/rollback ([#4197](#4197)) ([99af8fc](99af8fc)) * **zkstack:** fast fmt ([#4222](#4222)) ([d05c3dd](d05c3dd)) ### Bug Fixes * **consensus:** Debug page server port reuse ([#4273](#4273)) ([77d043f](77d043f)) * **eth-watch:** return internal errors from BatchRootProcessor ([#4285](#4285)) ([1f668e5](1f668e5)) * use FullPubdataBuilder for all pre-gateway batches ([#4265](#4265)) ([7c46480](7c46480)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: zksync-era-bot <[email protected]>

## What ❔ CI for interop integration tests broke recently as a result of the recent CI rework in: #4255. This PR defines a new (optional) secondary chain for integration tests, to be used for testing interop behavior, which defaults to the chain the tests are being run on.    ## Why ❔    ## Is this a breaking change? - [ ] Yes - [ ] No ## Operational changes   ## Checklist   - [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs). - [ ] Tests for the changes have been added / updated. - [ ] Documentation comments have been added / updated. - [ ] Code has been formatted via `zkstack dev fmt` and `zkstack dev lint`.

🤖 I have created a release *beep* *boop* --- ## [28.8.0](matter-labs/zksync-era@core-v28.7.0...core-v28.8.0) (2025-07-07) ### Features * **ci:** fast integration tests framework ([matter-labs#4255](matter-labs#4255)) ([a72cbd8](matter-labs@a72cbd8)) * **eigenda:** EigenDA V2 M0 ([matter-labs#3983](matter-labs#3983)) ([8302a81](matter-labs@8302a81)) * EN commit, prove, execute batch transactions verification and finality status ([matter-labs#4080](matter-labs#4080)) ([d06697d](matter-labs@d06697d)) * **en:** remove JSON RPC syncing ([matter-labs#4258](matter-labs#4258)) ([d194604](matter-labs@d194604)) * **state-keeper:** implement block commit/rollback ([matter-labs#4197](matter-labs#4197)) ([99af8fc](matter-labs@99af8fc)) * **zkstack:** fast fmt ([matter-labs#4222](matter-labs#4222)) ([d05c3dd](matter-labs@d05c3dd)) ### Bug Fixes * **consensus:** Debug page server port reuse ([matter-labs#4273](matter-labs#4273)) ([77d043f](matter-labs@77d043f)) * **eth-watch:** return internal errors from BatchRootProcessor ([matter-labs#4285](matter-labs#4285)) ([1f668e5](matter-labs@1f668e5)) * use FullPubdataBuilder for all pre-gateway batches ([matter-labs#4265](matter-labs#4265)) ([7c46480](matter-labs@7c46480)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: zksync-era-bot <[email protected]>

🤖 I have created a release *beep* *boop* --- ## [0.2.0](zkstack_cli-v0.1.2...zkstack_cli-v0.2.0) (2025-10-09) ### ⚠ BREAKING CHANGES * v29 upgrade testing & zkstack_cli changes ([#4332](#4332)) * **zkstack_cli:** Fix zkstack cli v28 script ([#3946](#3946)) * Remove old prover stack ([#3729](#3729)) * V27 update ([#3580](#3580)) ### Features * add dedicated TEE proof data handler module ([#3872](#3872)) ([ac64ee6](ac64ee6)) * add flag to save calldata ([#4471](#4471)) ([1ee5eac](1ee5eac)) * add integration test for DA migration ([#3944](#3944)) ([52aff1d](52aff1d)) * add prividium docker image support for block explorer ([#4127](#4127)) ([df9a433](df9a433)) * add prividium mode to zkstack explorer ([#4079](#4079)) ([c571914](c571914)) * Add proof manager contracts submodule ([#4189](#4189)) ([0c75985](0c75985)) * Add S3 implementation for object_store ([#3664](#3664)) ([a848927](a848927)) * Add Support for Protocol Version v28 ([#3821](#3821)) ([5419420](5419420)) * Adding 'rich_accounts' command to zkstack ([#3895](#3895)) ([50b72dc](50b72dc)) * **api:** Support Unix domain sockets for healthcheck server ([#4226](#4226)) ([b06bacb](b06bacb)) * **avail-client:** async blob dispatch ([#4010](#4010)) ([7a18647](7a18647)) * bump rustc to `nightly-2025-03-19` ([#3985](#3985)) ([d27390e](d27390e)) * **ci:** fast integration tests framework ([#4255](#4255)) ([a72cbd8](a72cbd8)) * **config:** Report config params in more ways ([#4126](#4126)) ([a78531c](a78531c)) * Configuration system PoC ([#3851](#3851)) ([7b449c2](7b449c2)) * **consensus:** Add consensus protocol versioning ([#3720](#3720)) ([d1b4308](d1b4308)) * **consensus:** Validator committee rotation ([#4014](#4014)) ([333efea](333efea)) * **contract_verifier:** add etherscan verification request support to the verifier api ([#3956](#3956)) ([87938b3](87938b3)) * **contract_verifier:** read compiler versions from cbor metadata if available ([#4002](#4002)) ([9bc20a4](9bc20a4)) * **contract-verifier:** add Etherscan contract verification ([#3609](#3609)) ([a4ea0f2](a4ea0f2)) * Draft v29 ([#3960](#3960)) ([91843a2](91843a2)) * **en:** Cache remote config for en ([#4367](#4367)) ([20bc4a8](20bc4a8)) * **en:** remove dependency on pubdata commitment mode ([#3826](#3826)) ([a0c78c0](a0c78c0)) * **en:** remove JSON RPC syncing ([#4258](#4258)) ([d194604](d194604)) * **en:** return back JSON RPC syncing ([#4344](#4344)) ([24b2990](24b2990)) * **en:** Use config system for env-based EN configuration ([#4104](#4104)) ([b706025](b706025)) * Eth proof manager sender ([#4266](#4266)) ([93b2086](93b2086)) * **eth_sender:** Add fast finalization into eth_tx_manager ([#4070](#4070)) ([c6b815d](c6b815d)) * **eth-sender:** set `from_addr` for non-blob txs ([#3898](#3898)) ([c699f8a](c699f8a)) * **gateway:** add checks that the server version is correct ([#3681](#3681)) ([659edaa](659edaa)) * **gateway:** Migration to Gateway ([#3654](#3654)) ([2858ba0](2858ba0)) * **gateway:** Requirement to stop L1->L2 transactions before v26 upgrade ([#3707](#3707)) ([0a095b7](0a095b7)) * **private-rpc:** improved compatibility with ethers library + tests ([#4046](#4046)) ([0e2e0d8](0e2e0d8)) * **private-rpc:** option to run private proxy in ZKStack CLI + moving Private Proxy files inside zksync-era repo ([#3919](#3919)) ([3b9307c](3b9307c)) * Proof data handler client ([#3874](#3874)) ([daf6f7b](daf6f7b)) * Proof manager watcher ([#4241](#4241)) ([6423b0d](6423b0d)) * Prover Cluster follow-up [#2](#2) ([#4001](#4001)) ([d8ed7f7](d8ed7f7)) * **prover:** --threads feature to WVGs ([#4291](#4291)) ([846f6e4](846f6e4)) * remove 2nd validator from command that prepares GW migration calldata ([#4331](#4331)) ([f81ad19](f81ad19)) * Remove feature flags for upgrades & v28.1 ([#4384](#4384)) ([fa7e679](fa7e679)) * Remove old prover stack ([#3729](#3729)) ([fbbdc76](fbbdc76)) * rework prover job identifiers ([#3888](#3888)) ([073326f](073326f)) * split zkstack cli command ([#4418](#4418)) ([36cd668](36cd668)) * update zkstack-cli for updated scripts ([#4481](#4481)) ([cc0a95b](cc0a95b)) * updating upgrade and migration config files for testnet and mainnet ([#4148](#4148)) ([6d11c2d](6d11c2d)) * Use JSON-RPC for core <> prover interaction ([#3626](#3626)) ([4e74730](4e74730)) * V27 update ([#3580](#3580)) ([9e18550](9e18550)) * **v27:** Use latest branch of release-v27 ([#3713](#3713)) ([6e1681e](6e1681e)) * v29 upgrade testing & zkstack_cli changes ([#4332](#4332)) ([9e4755e](9e4755e)) * **zkstack_cli:** Add function to track chain admin priority txs ([#3897](#3897)) ([fed2ca9](fed2ca9)) * **zkstack_cli:** finish enabling migrating chain from Gateway + remove the gateway feature flag ([#3924](#3924)) ([d091c90](d091c90)) * **zkstack_cli:** update gateway chain scripts ([#3852](#3852)) ([542c7a9](542c7a9)) * **zkstack-cli:** add prove & execute wallets ([#4477](#4477)) ([468cec9](468cec9)) * **zkstack:** Add zksync_os flag for ecosystem init ([#4456](#4456)) ([49e0529](49e0529)) * **zkstack:** Allow skipping build for contracts and server during ecosystem init ([#3697](#3697)) ([81abd84](81abd84)) * **zkstack:** Allow to run separate integration test suites ([d287725](d287725)) * **zkstack:** bump alloy ([#4502](#4502)) ([1a6ad60](1a6ad60)) * **zkstack:** Clarify prompt for funds check ([#3723](#3723)) ([a8ea520](a8ea520)) * **zkstack:** Deploy 2 ctms and deploy chain for each of them ([#4458](#4458)) ([5d2a7cd](5d2a7cd)) * **zkstack:** fast fmt ([#4222](#4222)) ([d05c3dd](d05c3dd)) * **zkstack:** Generating keys for compressor ([#4301](#4301)) ([b879b6f](b879b6f)) * **zkstack:** No genesis arg ([#4453](#4453)) ([9034246](9034246)) * **zkstack:** zkstack cli for v27 upgrade ([#3718](#3718)) ([91e04fb](91e04fb)) ### Bug Fixes * Changes to zkstack after testing migration from GW ([#3969](#3969)) ([b63e607](b63e607)) * **consensus:** Correctly handle error on SetValidatorSchedule ([#4147](#4147)) ([e169bd3](e169bd3)) * **consensus:** Debug page server port reuse ([#4273](#4273)) ([77d043f](77d043f)) * **consensus:** Update consensus dependencies ([#4186](#4186)) ([110a527](110a527)) * Fix security issues (bump dependencies) ([#3813](#3813)) ([c6def9c](c6def9c)) * fix zkstack cli upgrade features + remove redundant CI workflow ([#3975](#3975)) ([0f04134](0f04134)) * **gateway-migrator:** Properly handle unknown settlement layer ([#3961](#3961)) ([b43e315](b43e315)) * make proof data handler backwards compatible ([#3767](#3767)) ([bdbbaaa](bdbbaaa)) * remove zksync_contracts dep from zkstack_cli ([#4400](#4400)) ([7bcf275](7bcf275)) * target_rpc address in private-rpc docker-compose ([#4062](#4062)) ([b69819b](b69819b)) * **upgrades:** Read all skipped events ([#4504](#4504)) ([1de5f63](1de5f63)) * **zkstack_cli:** Fix zkstack cli v28 script ([#3946](#3946)) ([0a3d13f](0a3d13f)) * **zkstack:** add `--locked` to `cargo sqlx prepare` ([#3300](#3300)) ([a98b1c8](a98b1c8)) * **zkstack:** Add param for setting da validation pair ([#4150](#4150)) ([47e2517](47e2517)) * **zkstack:** Allow to use chain config if ecosystem is redundant ([#4236](#4236)) ([066b3b1](066b3b1)) * **zkstack:** Fix prover run and shell ([#4422](#4422)) ([eae38e2](eae38e2)) * **zkstack:** fixes for private rpc in zkstack cli ([#4012](#4012)) ([be68210](be68210)) * **zkstack:** Generating compressor keys flow fix ([#4343](#4343)) ([4cf2f67](4cf2f67)) * **zkstack:** make GW migration script compatible with pre-v29 ([#4346](#4346)) ([7ba2cf3](7ba2cf3)) * **zkstack:** make proving network contracts config optional ([#4263](#4263)) ([3ebad3b](3ebad3b)) * **zkstack:** Migration params ([#4369](#4369)) ([8597c32](8597c32)) * **zkstack:** Use `latest` for prover component by default ([#4120](#4120)) ([91af29d](91af29d)) ### Performance Improvements * **db:** use copy in `insert_initial_writes` ([#3899](#3899)) ([c6f1598](c6f1598)) * Instrumentation for Jemalloc (pt. 2) ([#4204](#4204)) ([5e0bd65](5e0bd65)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: zksync-era-bot <[email protected]>

github-advanced-security bot found potential problems Jun 28, 2025

View reviewed changes

tomg10 changed the title ~~Fast integration tests experiments~~ feat(ci): fast integration tests experiments Jun 29, 2025

github-advanced-security bot found potential problems Jun 29, 2025

View reviewed changes

core/tests/highlevel-test-tools/src/execute-command.ts Fixed Show fixed Hide fixed

core/tests/highlevel-test-tools/src/run-integration-tests.ts Fixed Show fixed Hide fixed

core/tests/highlevel-test-tools/src/start-server.ts Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems Jun 29, 2025

View reviewed changes

core/tests/highlevel-test-tools/src/start-server.ts Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems Jun 30, 2025

View reviewed changes

core/tests/highlevel-test-tools/src/gateway.ts Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems Jul 1, 2025

View reviewed changes

core/tests/highlevel-test-tools/src/start-external-node.ts Fixed Show fixed Hide fixed

github-advanced-security bot found potential problems Jul 1, 2025

View reviewed changes

core/tests/highlevel-test-tools/src/start-external-node.ts Fixed Show fixed Hide fixed

tomg10 marked this pull request as ready for review July 1, 2025 22:34

tomg10 requested a review from a team as a code owner July 1, 2025 22:34

tomg10 requested review from alexandrst88, artmakh, hatemosphere, iluwaa, onyxet, otani88 and yorik July 1, 2025 22:34

github-advanced-security bot found potential problems Jul 2, 2025

View reviewed changes

tomg10 changed the title ~~feat(ci): fast integration tests experiments~~ feat(ci): fast integration tests framework Jul 2, 2025

popzxc reviewed Jul 2, 2025

View reviewed changes

core/tests/highlevel-test-tools/package.json Outdated Show resolved Hide resolved

bin/build_and_init_ecosystem Show resolved Hide resolved

core/tests/highlevel-test-tools/global-setup.ts Show resolved Hide resolved

Deniallugo reviewed Jul 2, 2025

View reviewed changes

tomg10 added 8 commits July 3, 2025 19:21

split chain initialization

c6068f8

fix

b6f4778

fmt

35bc2a4

test

7d322ff

test

9964527

fix

ee0ece9

fix

4458217

fix

c6e3ba6

tomg10 added 4 commits July 4, 2025 02:27

fix

f94bf5e

fix

af1c4d6

add missing init_test_wallet

e922dbf

revert setup-env for building contracts

1bcdb2c

RomanBrodetski previously approved these changes Jul 4, 2025

View reviewed changes

fix command

ead815d

tomg10 dismissed RomanBrodetski’s stale review via ead815d July 4, 2025 16:35

tomg10 added 3 commits July 7, 2025 10:24

fix command

98a5892

lint

9889930

fmt

079a9fd

RomanBrodetski approved these changes Jul 7, 2025

View reviewed changes

tomg10 enabled auto-merge July 7, 2025 10:47

otani88 approved these changes Jul 7, 2025

View reviewed changes

tomg10 added this pull request to the merge queue Jul 7, 2025

artmakh approved these changes Jul 7, 2025

View reviewed changes

Merged via the queue into main with commit a72cbd8 Jul 7, 2025
50 checks passed

tomg10 deleted the fast-integration-tests-experiments branch July 7, 2025 11:35

This was referenced Jul 7, 2025

chore(main): release core 28.8.0 #4260

Merged

chore(main): release zkstack_cli 0.2.0 #3086

Merged

zkzoomer pushed a commit that referenced this pull request Jul 7, 2025

feat(ci): fast integration tests framework (#4255)

7e97586

zkzoomer mentioned this pull request Jul 9, 2025

fix: fix interop integration tests #4300

Merged

6 tasks

zkzoomer mentioned this pull request Jul 16, 2025

fix: prevent uncommitted chain migrating from gw #4314

Merged

6 tasks

jishnundth pushed a commit to jishnundth/zksync-era that referenced this pull request Oct 6, 2025

feat(ci): fast integration tests framework (matter-labs#4255)

b5bcb8c

feat(ci): fast integration tests framework #4255

feat(ci): fast integration tests framework #4255

Uh oh!

Conversation

tomg10 commented Jun 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Check warning

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Check warning

Uh oh!

Uh oh!

Check warning

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tomg10 commented Jun 28, 2025 •

edited

Loading