refactor fhevm cli runtime model

Author: Eikix

test-suite/fhevm/ARCHITECTURE.md

@@ -10,12 +10,15 @@ flowchart TD
   B --> B2["latest-release: latest stable release"]
   B --> B3["sha: exact repo-owned SHA on main, fail if any package tag is missing or if it predates 803f104"]
   B --> B4["devnet/testnet/mainnet: GitOps bundles"]
-  B1 --> C["Lock resolved bundle"]
+  B1 --> C["apply *_VERSION env overrides"]
   B2 --> C
   B3 --> C
   B4 --> C
 
-  C --> E["3. generate runtime files under .fhevm"]
+  C --> C1["Lock resolved bundle"]
+  C1 --> C2["apply coprocessor scenario or --override coprocessor shorthand"]
+
+  C2 --> E["3. generate runtime files under .fhevm"]
   E --> E1["env/"]
   E --> E2["compose/"]
   E --> E3["locks/"]
@@ -36,8 +39,8 @@ flowchart TD
   F10 --> F11["14. relayer"]
   F11 --> F12["15. test-suite"]
 
-  G["Local overrides (group or runtime service)"] --> E
-  H["Multicopro topology + per-instance overrides"] --> E
+  G["Local overrides (group or runtime service)"] --> C2
+  H["Scenario-driven coprocessor topology"] --> C2
   I["Compatibility policy"] --> E
   I --> F8
 
@@ -71,8 +74,18 @@ the SHA-tagged images for every component built from the PR.
 - Version selection is explicit. The CLI does not silently use a vague "latest".
 - `latest-main` is modern-only by construction. If no complete bundle exists after the floor SHA, resolution fails.
 - The resolved bundle is printed and locked before the real boot continues.
+- Runtime precedence is fixed: bundle -> `*_VERSION` env overrides -> coprocessor scenario/shorthand -> generated runtime files.
 - `.fhevm` is the only mutable runtime area owned by the CLI.
+- Tracked inputs are split by role:
+  - compose templates: `docker-compose/*.yml`
+  - env templates: `templates/env/.env.*`
+  - relayer template config: `templates/config/relayer.yaml`
+  - static config: `static/config/kms-core/config.toml`, `static/config/prometheus/prometheus.yml`
+  - scenario inputs: `scenarios/*.yaml`
+- `src/runtime-plan.ts` resolves the final coprocessor/runtime shape consumed by regeneration.
+- `src/render-env.ts`, `src/render-config.ts`, and `src/render-compose.ts` are the only rendering layers.
 - Discovery is not terminal output only. It feeds env regeneration before dependent services start.
 - Resume is step-based via `state.json`, not "rerun the bash ritual and hope".
-- `upgrade` is intentionally narrow: it only rebuilds and restarts active runtime override groups.
+- Tracked compose files are the default runtime truth. `.fhevm/compose` only contains generated overrides for coprocessor topology and active local-override components.
+- `upgrade` is intentionally narrow: it only rebuilds and restarts active runtime override groups or local coprocessor scenario instances.
 - `up --dry-run` exercises the same target-aware resolve and preflight path without mutating runtime state.

test-suite/fhevm/README.md

@@ -6,7 +6,7 @@ It exists for three workflows:
 
 - run a known stack target locally
 - swap in local changes for one repo-owned group
-- run multicopro topologies with deterministic generated state
+- run consensus/matrix coprocessor scenarios with deterministic generated state
 
 The CLI owns all mutable runtime state under `.fhevm/`. Tracked compose and env files stay as templates.
 
@@ -30,13 +30,53 @@ bun test
 
 - `up` resolves a target bundle, runs preflight, generates `.fhevm`, and boots the stack
 - `up --dry-run` runs the same resolve and preflight path without mutating runtime state
+- `up --scenario <file>` applies an explicit coprocessor consensus scenario on top of the resolved bundle
+- `up --override coprocessor` is the fast local-dev shorthand for a one-instance local coprocessor scenario
 - `test` runs against the current stack; it does not recompile contracts. `--parallel` runs tests in parallel (auto for `operators`)
 - `logs` follows container output; `--no-follow` prints the tail and exits
 - `pause` / `unpause` pauses or unpauses host or gateway contracts
 - `down` stops the stack
 - `clean` removes CLI-owned runtime state
 - `clean --images` also removes CLI-owned local override images
 
+## Ownership Model
+
+There are four kinds of inputs/runtime artifacts:
+
+- tracked compose templates: `docker-compose/*.yml`
+- tracked env templates: `templates/env/.env.*`
+- tracked config:
+  - relayer template input: `templates/config/relayer.yaml`
+  - static mounted config: `static/config/kms-core/config.toml`, `static/config/prometheus/prometheus.yml`
+- tracked scenario inputs: `scenarios/*.yaml`
+
+Generated runtime artifacts always live under `.fhevm/`:
+
+- `.fhevm/env/*.env`
+- `.fhevm/compose/*.yml` for generated runtime overrides only
+- `.fhevm/config/relayer.yaml`
+- `.fhevm/addresses/*`
+- `.fhevm/locks/*`
+- `.fhevm/state.json`
+
+Tracked compose files are the default runtime truth. `.fhevm/compose` only holds generated overrides when runtime structure or local-image policy actually changes, with coprocessor topology as the only structural expansion.
+
+The code follows the same split:
+
+- `src/runtime-plan.ts`: resolve one runtime plan from bundle + env overrides + scenario/shorthand
+- `src/render-env.ts`: render runtime env maps
+- `src/render-config.ts`: render generated config files
+- `src/render-compose.ts`: render compose overlays, with coprocessor topology as the only structural exception
+
+## Resolution Order
+
+Runtime resolution is intentionally fixed:
+
+1. Resolve the base bundle from `--target`, `--sha`, or `--lock-file`
+2. Apply matching `*_VERSION` environment overrides
+3. Apply either `--scenario <file>` or the `--override coprocessor` shorthand
+4. Materialize generated env/config/compose state under `.fhevm/`
+
 ## Targets
 
 - `latest-release`: latest stable fhevm release plus checked-in companion defaults
@@ -160,7 +200,7 @@ The matrix has four sections:
 | `externalDefaults` | Pinned versions for non-workspace components | modern relayer SHA |
 | `anchors` | Git history reference points | simple-ACL cutover commit |
 
-CI workflows read these values via `./fhevm-cli compat-defaults` instead of hardcoding them.
+CI workflows read `externalDefaults` and `anchors` via `./fhevm-cli compat-defaults` instead of hardcoding them.
 
 ### How to update
 
@@ -187,7 +227,7 @@ When the minimum supported version passes the threshold, delete the `legacyShims
 ./fhevm-cli up --target latest-release --resume --from-step relayer
 ./fhevm-cli up --target latest-release --override coprocessor
 ./fhevm-cli up --target latest-release --override coprocessor:host-listener,tfhe-worker
-./fhevm-cli up --target latest-release --coprocessors 2 --threshold 2
+./fhevm-cli up --target latest-release --scenario ./scenarios/two-of-two.yaml
 ./fhevm-cli upgrade coprocessor
 
 ./fhevm-cli status
@@ -222,6 +262,8 @@ Supported groups:
 ./fhevm-cli up --target latest-release --override coprocessor
 ```
 
+For `coprocessor`, this is also the shorthand local-dev scenario: one coprocessor instance, threshold `1`, source mode `local`.
+
 ### Override specific runtime services
 
 Runtime override groups also support per-service filtering:
@@ -284,27 +326,31 @@ If a runtime override is already active and you only want to rebuild and restart
 ./fhevm-cli upgrade coprocessor
 ```
 
-`upgrade` only supports active runtime override groups: `coprocessor`, `kms-connector`, and `test-suite`. It rebuilds and restarts runtime services only; one-shot DB migration containers are not rerun.
+`upgrade` only supports active runtime override groups: `coprocessor`, `kms-connector`, and `test-suite`. For `coprocessor`, it rebuilds only the local coprocessor instances from the active shorthand/scenario state. One-shot DB migration containers are not rerun.
 
 ## Dropped Convenience Commands
 
 - `smoke`: use explicit `up ...` plus `test ...`
 - `test debug`: use `docker exec -it fhevm-test-suite-e2e-debug sh`
 
-## Multicopro
+## Coprocessor Scenarios
+
+Use `--scenario <file>` for consensus and rollout matrices. The file is the source of truth for:
+
+- coprocessor count and threshold
+- per-instance source mode: `inherit`, `registry`, or `local`
+- per-instance env overrides
+- per-instance runtime args
 
-Example:
+Examples:
 
 ```sh
-./fhevm-cli up \
-  --target latest-release \
-  --coprocessors 2 \
-  --threshold 2 \
-  --instance-env 1:OTEL_SERVICE_NAME=coprocessor-1-local \
-  --instance-arg '1:tfhe-worker=--coprocessor-fhe-threads=4'
+./fhevm-cli up --target latest-release --scenario ./scenarios/two-of-two.yaml
+./fhevm-cli up --target latest-release --scenario ./scenarios/one-registry-outlier.yaml
+./fhevm-cli up --target latest-release --scenario ./scenarios/one-local-outlier.yaml
 ```
 
-Generated env, compose overlays, addresses, locks, and state all live under `.fhevm/`.
+`--scenario` cannot be combined with `--override coprocessor`. Keep `--override coprocessor` for the fast local e2e loop; use scenarios when you need an explicit consensus matrix.
 
 ## Runtime State
 
@@ -316,4 +362,4 @@ The CLI owns:
 - `.fhevm/compose/`
 - `.fhevm/addresses/`
 
-`status` shows the active stack state and any CLI-owned local build images.
+`status` shows the active stack state, the active scenario origin when present, and any CLI-owned local build images.

test-suite/fhevm/bun.lock

@@ -26,7 +26,7 @@ services:
       cache_from:
         - ${FHEVM_CACHE_FROM_COPROCESSOR:-type=gha}
     env_file:
-      - ../env/staging/.env.coprocessor.local
+      - ../../../.fhevm/env/coprocessor.env
     environment:
       - KEY_ID=${FHE_KEY_ID}
     command:
@@ -45,7 +45,7 @@ services:
       cache_from:
         - ${FHEVM_CACHE_FROM_COPROCESSOR:-type=gha}
     env_file:
-      - ../env/staging/.env.coprocessor.local
+      - ../../../.fhevm/env/coprocessor.env
     command:
       - host_listener
       - --database-url=${DATABASE_URL}
@@ -68,7 +68,7 @@ services:
       cache_from:
         - ${FHEVM_CACHE_FROM_COPROCESSOR:-type=gha}
     env_file:
-      - ../env/staging/.env.coprocessor.local
+      - ../../../.fhevm/env/coprocessor.env
     command:
       - host_listener_poller
       - --database-url=${DATABASE_URL}
@@ -95,7 +95,7 @@ services:
       timeout: 5s
       retries: 3
     env_file:
-      - ../env/staging/.env.coprocessor.local
+      - ../../../.fhevm/env/coprocessor.env
     command:
       - gw_listener
       - --database-url=${DATABASE_URL}
@@ -124,7 +124,7 @@ services:
       cache_to:
         - ${FHEVM_CACHE_TO_COPROCESSOR:-type=gha,mode=max}
     env_file:
-      - ../env/staging/.env.coprocessor.local
+      - ../../../.fhevm/env/coprocessor.env
     command:
       - tfhe_worker
       - --run-bg-worker
@@ -149,7 +149,7 @@ services:
       cache_from:
         - ${FHEVM_CACHE_FROM_COPROCESSOR:-type=gha}
     env_file:
-      - ../env/staging/.env.coprocessor.local
+      - ../../../.fhevm/env/coprocessor.env
     command:
       - zkproof_worker
       - --database-url=${DATABASE_URL}
@@ -172,7 +172,7 @@ services:
       cache_from:
         - ${FHEVM_CACHE_FROM_COPROCESSOR:-type=gha}
     env_file:
-      - ../env/staging/.env.coprocessor.local
+      - ../../../.fhevm/env/coprocessor.env
     command:
       - sns_worker
       - --database-url=${DATABASE_URL}
@@ -207,7 +207,7 @@ services:
       cache_from:
         - ${FHEVM_CACHE_FROM_COPROCESSOR:-type=gha}
     env_file:
-      - ../env/staging/.env.coprocessor.local
+      - ../../../.fhevm/env/coprocessor.env
     command:
       - transaction_sender
       - --database-url=${DATABASE_URL}

test-suite/fhevm/config/core-client/config.toml

@@ -4,7 +4,7 @@ services:
     container_name: kms-core
     image: ghcr.io/zama-ai/kms/core-service:${CORE_VERSION}
     env_file:
-      - ../env/staging/.env.core.local
+      - ../../../.fhevm/env/core.env
     entrypoint:
       - /bin/sh
       - -c
@@ -18,15 +18,11 @@ services:
         kms-server --config-file config/config.toml
     volumes:
       - fhevm_minio_secrets:/minio_secrets
-      - ../config/kms-core/config.toml:/app/kms/core/service/config/config.toml
+      - ../static/config/kms-core/config.toml:/app/kms/core/service/config/config.toml
     ports:
       - "50051:50051"
     healthcheck:
-      test: "grpc_health_probe --addr=localhost:50051"
-      interval: 1s
-      timeout: 1s
-      retries: 5
-      start_period: 1s
+      disable: true
 
 volumes:
   fhevm_minio_secrets:

test-suite/fhevm/docker-compose/coprocessor-docker-compose.yml

@@ -8,7 +8,7 @@ services:
       - -c
       - max_connections=500
     env_file:
-      - ../env/staging/.env.database.local
+      - ../../../.fhevm/env/database.env
     ports:
       - '5432:5432'
     healthcheck:

test-suite/fhevm/docker-compose/core-docker-compose.yml

@@ -10,7 +10,7 @@ services:
       cache_to:
         - ${FHEVM_CACHE_TO_GATEWAY_DEPLOY_MOCKED_ZAMA_OFT:-type=gha,mode=max}
     env_file:
-      - ../env/staging/.env.gateway-mocked-payment.local
+      - ../../../.fhevm/env/gateway-mocked-payment.env
     command:
       - npx hardhat task:deployMockedZamaOFT
 
@@ -26,9 +26,9 @@ services:
       cache_to:
         - ${FHEVM_CACHE_TO_GATEWAY_SET_RELAYER_MOCKED_PAYMENT:-type=gha,mode=max}
     env_file:
-      - ../env/staging/.env.gateway-mocked-payment.local
+      - ../../../.fhevm/env/gateway-mocked-payment.env
     command:
       - npx hardhat task:setTxSenderMockedPayment
     depends_on:
       gateway-deploy-mocked-zama-oft:
-        condition: service_completed_successfully
+        condition: service_completed_successfully

test-suite/fhevm/docker-compose/database-docker-compose.yml

@@ -3,7 +3,7 @@ services:
     container_name: gateway-node
     image: ghcr.io/foundry-rs/foundry:v1.3.5
     env_file:
-      - ../env/staging/.env.gateway-node.local
+      - ../../../.fhevm/env/gateway-node.env
     entrypoint:
       - anvil
       - --block-time

test-suite/fhevm/docker-compose/gateway-mocked-payment-docker-compose.yml

@@ -10,11 +10,8 @@ services:
       cache_to:
         - ${FHEVM_CACHE_TO_GATEWAY_SC_PAUSE:-type=gha,mode=max}
     env_file:
-      - ../env/staging/.env.gateway-sc.local
+      - ../../../.fhevm/env/gateway-sc.env
     command:
       - npx hardhat compile && npx hardhat task:pauseAllGatewayContracts
     volumes:
-      - addresses-volume:/app/addresses  # workdir in gateway's Dockerfile is /app
-
-volumes:
-  addresses-volume:
+      - ../../../.fhevm/addresses/gateway:/app/addresses  # workdir in gateway's Dockerfile is /app