feat(test-suite): replace fhevm bash cli with Bun runtime by Eikix · Pull Request #2066 · zama-ai/fhevm

Eikix · 2026-03-09T09:40:21Z

Closes https://github.com/zama-ai/fhevm-internal/issues/1042

Summary

Replace the legacy test-suite/fhevm/fhevm-cli bash workflow with a Bun-native TypeScript CLI.

The new CLI makes target selection, runtime generation, resume, local builds, upgrade flows, and multicopro scenarios explicit and reproducible instead of relying on bash defaults and workflow-specific behavior.

User-facing changes

boot explicit targets:
- latest-supported
- latest-main
- sha --sha <git-sha>
- devnet
- testnet
- mainnet
treat latest-supported as a tracked maintained bundle profile:
- test-suite/fhevm/profiles/latest-supported.json
keep latest-main as the moving mainline baseline:
- newest complete repo-owned main SHA bundle at or after the simple-ACL floor
- non-repo companions come from the mainline defaults in src/presets.ts
lock every resolved bundle under .fhevm/locks
persist runtime state under .fhevm/state.json
support full-group local overrides for:
- coprocessor
- kms-connector
- gateway-contracts
- host-contracts
- test-suite
support per-service overrides only for runtime groups:
- coprocessor
- kms-connector
- test-suite
support --build as the full local-workspace path:
- normal stacks: full local workspace, including coprocessor
- scenario runs: full local workspace except coprocessor, which stays scenario-driven
support upgrade <group> for active runtime overrides:
- coprocessor
- kms-connector
- test-suite
replace the old multicopro CLI flags with explicit scenario files:
- fhevm-cli up --scenario ./scenarios/two-of-two.yaml
keep --override coprocessor as the fast local-dev shorthand
keep deploy as an alias for up
remove doctor in favor of up --dry-run

Architecture

Tracked inputs are now split by ownership:

test-suite/fhevm/docker-compose/: base compose truth
test-suite/fhevm/templates/: tracked templates rendered by the CLI
test-suite/fhevm/static/: tracked config mounted as-is
test-suite/fhevm/scenarios/: checked-in consensus/matrix inputs
test-suite/fhevm/profiles/: tracked maintained bundle profiles

Generated runtime output lives under .fhevm/:

.fhevm/env/
.fhevm/config/
.fhevm/compose/
.fhevm/addresses/
.fhevm/locks/
.fhevm/state.json

The runtime flow is now:

resolve the target/profile bundle
apply version env overrides
resolve the coprocessor scenario or --override coprocessor shorthand
build a normalized runtime plan
render env/config/compose artifacts
run the step-based pipeline with resume support

The code is split around that model:

src/resolve.ts
src/scenario.ts
src/runtime-plan.ts
src/render-env.ts
src/render-config.ts
src/render-compose.ts
src/pipeline.ts

Tracked compose files are the default runtime truth. .fhevm/compose/ now contains selective overrides instead of full generated replacements for every component. Coprocessor topology remains the structural exception.

Compatibility and targeting

latest-main and sha are hard-cut to the simple-ACL era and fail before the supported floor
exact sha resolution requires a complete repo-owned image set and a commit that exists on main
latest-main resolves repo-owned services from main; src/presets.ts supplies the non-repo companion baseline for the common mainline path
src/compat.ts is kept focused on cross-era protection:
- known-invalid bundle pairs
- legacy runtime shims
- target resolution anchors

CI contract

reusable e2e now always boots latest-main with the checked-in two-of-two scenario
manual workflow dispatch defaults to build=true, so a human dispatch naturally validates the checked out branch from source
merge-queue orchestration passes build=false explicitly and overlays repo-owned *_VERSION env vars only for components whose Docker build succeeded
skipped component builds leave those overrides unset, so they naturally stay on the latest-main baseline
merge queue fails before e2e if a required repo-owned build reports a non-skipped failure
operators e2e uses the same latest-main baseline and scenario model

Quality and reliability improvements

replace the monolithic bash runtime with typed services and tests
make runtime artifact regeneration deterministic for resume, upgrade, and teardown
make Docker/preflight failures surface clearly instead of degrading into misleading output
fix lock-file boots, exact SHA handling, runtime error rendering, and stale state handling
add scenario parsing/validation for consensus workflows
keep state persistence normalized by deriving topology from scenario on load instead of persisting both authorities
port the ciphertext-drift path off bash into shared TS/Bun code

Validation

Verified on the current branch state:

bun run check
bun test (322 tests)
repeated local QA and live-stack validation across:
- baseline lifecycle
- resume / regenerate paths
- local override flows
- full local-workspace --build dry-runs
- upgrade coprocessor
- scenario-driven multicopro boot paths
- latest-main dry-runs and scenario dry-runs

Notes

latest-supported is now a tracked maintained profile, not a dynamic “latest release” resolver.
latest-main is the moving baseline for PR validation and local mainline testing.
--scenario remains the explicit consensus/matrix interface.
--override coprocessor remains the fast local e2e path.
--build is the full local-workspace path.

coprocessor/fhevm-engine/Dockerfile.workspace

.github/workflows/test-suite-e2e-cli-tests.yml

.github/workflows/test-suite-e2e-operators-tests.yml

.github/workflows/test-suite-e2e-tests.yml

.github/workflows/test-suite-e2e-operators-tests.yml

.github/workflows/test-suite-e2e-tests.yml

mergify · 2026-03-09T18:28:54Z

🧪 CI Insights

Here's what we observed from your CI run for 89296fd.

❌ Job Failures

Pipeline	Job	Health on `main`	Retries	🔍 CI Insights	📄 Logs
`coprocessor-cargo-test`	`coprocessor-cargo-test/cargo-tests (bpr)`		`0`	View	View

eudelins-zama

Just a small note to say it would be nice to merge this feature branch first to avoid handling conflicts with this one 😇

Eikix · 2026-03-10T09:28:45Z

@claude /pr-review

PanGan21 · 2026-03-10T12:53:08Z

@cla-bot check

cla-bot · 2026-03-10T12:53:13Z

The cla-bot has been summoned, and re-checked this pull request!

Eikix · 2026-03-10T16:36:52Z

Opened a sacrificial draft PR to exercise the merge-queue-style E2E orchestration on the same head commit as this PR: #2087

It is draft-only and should not be merged. Once we have the workflow signal, we can close it.

.github/workflows/test-suite-e2e-operators-tests.yml

test-suite/fhevm/scenarios/two-of-two.yaml

Eikix · 2026-03-19T11:10:59Z

@claude /pr-review

claude · 2026-03-19T11:11:50Z

Claude timed out while processing the request. — View run

Eikix · 2026-03-19T13:56:32Z

.github/workflows/test-suite-e2e-tests.yml

      - name: Show logs on test failure
        working-directory: test-suite/fhevm
-        if: always()
+        if: failure()


Go back to always()

Eikix · 2026-03-19T14:14:54Z

.github/workflows/test-suite-e2e-operators-tests.yml

-          ZKPROOF_WORKER_VERSION: ${{ inputs.zkproof_worker_version }}
        run: |
-          ./fhevm-cli deploy --coprocessors 2 --coprocessor-threshold 2
+          ./fhevm-cli up --target latest-main --scenario ./scenarios/two-of-two.yaml


can we use the short form here ./fhevm-cli up --target latest-main --scenario two-of-two

Eikix · 2026-03-19T14:15:03Z

.github/workflows/test-suite-e2e-operators-tests.yml

      - name: Show logs on test failure
        working-directory: test-suite/fhevm
-        if: always()
+        if: failure()


use always()

cla-bot bot added the cla-signed label Mar 9, 2026

Eikix commented Mar 9, 2026

View reviewed changes

coprocessor/fhevm-engine/Dockerfile.workspace Show resolved Hide resolved

github-advanced-security bot found potential problems Mar 9, 2026

View reviewed changes

.github/workflows/test-suite-e2e-cli-tests.yml Fixed Show fixed Hide fixed

zama-ai deleted a comment from claude bot Mar 9, 2026

github-advanced-security bot found potential problems Mar 9, 2026

View reviewed changes

Eikix force-pushed the codex/bun-fhevm-cli branch from 7408594 to a651dc5 Compare March 9, 2026 19:45

Eikix marked this pull request as ready for review March 10, 2026 09:13

Eikix requested review from a team as code owners March 10, 2026 09:13

eudelins-zama reviewed Mar 10, 2026

View reviewed changes

cla-bot bot removed the cla-signed label Mar 10, 2026

cla-bot bot added the cla-signed label Mar 10, 2026

zama-ai deleted a comment from claude bot Mar 10, 2026

zama-ai deleted a comment from cla-bot bot Mar 10, 2026

zama-ai deleted a comment from claude bot Mar 10, 2026

Eikix force-pushed the codex/bun-fhevm-cli branch from 4b82c14 to 8fce5b0 Compare March 10, 2026 15:14

zama-ai deleted a comment from claude bot Mar 10, 2026

Eikix mentioned this pull request Mar 10, 2026

chore(test-suite): sacrificial merge-queue validation for #2066 #2087

Closed

Eikix force-pushed the codex/bun-fhevm-cli branch from aab6a5e to fcff473 Compare March 11, 2026 09:02

Eikix commented Mar 11, 2026

View reviewed changes

.github/workflows/test-suite-e2e-operators-tests.yml Outdated Show resolved Hide resolved

Eikix commented Mar 18, 2026

View reviewed changes

test-suite/fhevm/scenarios/two-of-two.yaml Show resolved Hide resolved

Eikix marked this pull request as ready for review March 18, 2026 17:02

Eikix added 11 commits March 18, 2026 18:06

Trim unused fhevm CLI exports

738b167

Drop QA agent prompt from PR

dbfeeab

Drop unused fhevm scenarios

c6a60a4

Sync fhevm CLI docs with current scenario set

0f78c9c

Fix forced rebuilds in fhevm upgrade

9387b15

Fix fhevm CLI lifecycle and drift guards

2fa9f2a

Fix fhevm test command failure handling

cb67f34

Tighten fhevm CLI review follow-ups

899718f

Tighten fhevm lock and CI guards

a8d9a11

Improve fhevm CLI scenario and lifecycle UX

2af0175

Refine fhevm runtime state and logs

b5be714

Eikix added 4 commits March 19, 2026 12:22

ci(test-suite): restore release orchestrate gating

8e9cbaf

fix(test-suite): tighten compat and upgrade guards

168eb7c

refactor(test-suite): drop dead no-relayer flag

3894613

fix(ci): run orchestrated e2e on merge-queue PRs

1c31abc

Eikix marked this pull request as draft March 19, 2026 13:00

Eikix added 3 commits March 19, 2026 14:39

refactor(test-suite): hard cutover to lean bun cli

5957258

chore(test-suite): trim dead exports and refresh docs

38db8af

fix(test-suite): tighten lean cli edge cases

7349c1a

Eikix commented Mar 19, 2026

View reviewed changes

Eikix added 2 commits March 19, 2026 15:31

refactor(test-suite): flatten cli source layout

d14fab9

fix(test-suite): tighten lean cli review findings

888881a

Eikix mentioned this pull request Mar 19, 2026

refactor(test-suite): replace fhevm bash cli with lean Bun TS #2134

Open

Eikix closed this Mar 19, 2026

Conversation

Eikix commented Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

User-facing changes

Architecture

Compatibility and targeting

CI contract

Quality and reliability improvements

Validation

Notes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mergify bot commented Mar 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🧪 CI Insights

❌ Job Failures

Uh oh!

eudelins-zama left a comment

Choose a reason for hiding this comment

Uh oh!

Eikix commented Mar 10, 2026

Uh oh!

PanGan21 commented Mar 10, 2026

Uh oh!

cla-bot bot commented Mar 10, 2026

Uh oh!

Eikix commented Mar 10, 2026

Uh oh!

Uh oh!

Uh oh!

Eikix commented Mar 19, 2026

Uh oh!

claude bot commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Eikix Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

Eikix Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

Eikix Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Eikix commented Mar 9, 2026 •

edited

Loading

mergify bot commented Mar 9, 2026 •

edited

Loading

claude bot commented Mar 19, 2026 •

edited

Loading