feat(common): setup KMS materials in test-suite and coprocessor#876
Merged
isaacdecoded merged 35 commits intofeat/keygen_via_gatewayfrom Sep 30, 2025
Merged
Conversation
melanciani
changed the title
melanciani
force-pushed
the
feat/keygen_via_gateway
branch
from
f3c4359 to
1160d00
Compare
isaacdecoded
force-pushed
the
feat/keygen_via_gateway
branch
from
1160d00 to
801d1f7
Compare
isaacdecoded
force-pushed
the
isaac/383/test/fhe-keygen-through-gateway
branch
from
cfb8426 to
228b3f6
Compare
There was a problem hiding this comment.
Pull Request Overview
This PR refactors the FHEVM stack deployment for keygen end-to-end testing by restructuring the deployment process, reorganizing component dependencies, and updating configuration properties to use more generic terminology.
- Restructures the FHEVM stack deployment to separate node and contract deployment phases
- Updates KMS-related property naming from "keygen" to "kmsGen" and "s3BucketUrl" to "storageUrl" for clarity
- Adds a new script for setting up KMS signer addresses used in Gateway and Host contracts
Reviewed Changes
Copilot reviewed 37 out of 39 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| test-suite/fhevm/scripts/update-kms-keys.sh | Comments out key retrieval and environment update logic for restructured deployment |
| test-suite/fhevm/scripts/setup-kms-signer-address.sh | Adds new script to configure KMS signer addresses for Gateway/Host contracts |
| test-suite/fhevm/scripts/deploy-fhevm-stack.sh | Restructures deployment flow with separated node/contract phases and updated component list |
| test-suite/fhevm/fhevm-cli | Updates service version configurations and switches to specific commit hashes |
| test-suite/fhevm/env/staging/.env.* | Adds new environment files for separated host/gateway nodes and database services |
| test-suite/fhevm/docker-compose/*.yml | Restructures Docker Compose files to separate node services from contract deployments |
| gateway-contracts/test/utils/contracts.ts | Renames kmsNodeS3BucketUrls to kmsNodeStorageUrls for consistency |
| gateway-contracts/test/mocks/mocks.ts | Updates mock property names from s3BucketUrl to storageUrl |
| gateway-contracts/test/*.ts | Updates test references from keygen to kmsGen threshold terminology |
| gateway-contracts/tasks/deployment/contracts.ts | Updates deployment task to use new kmsGenThreshold and storageUrl properties |
| gateway-contracts/selectors.txt | Updates function/event selectors for renamed threshold methods |
| gateway-contracts/rust_bindings/*.rs | Updates Rust binding names to reflect storageUrl and kmsGenThreshold changes |
| gateway-contracts/hardhat.config.ts | Adds generateFheKeys task import |
| gateway-contracts/docs/getting-started/**/*.md | Updates documentation for new threshold and storage URL terminology |
| gateway-contracts/docker-compose.yml | Adds trigger-crsgen service for CRS generation testing |
| gateway-contracts/contracts/**/*.sol | Updates contract interfaces and implementations for new naming conventions |
| gateway-contracts/.env.example | Updates environment variable examples with new naming |
| .github/workflows/gateway-contracts-deployment-tests.yml | Adds CRS generation triggering validation to deployment tests |
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
rudy-6-4
reviewed
Sep 26, 2025
rudy-6-4
reviewed
Sep 26, 2025
rudy-6-4
approved these changes
Sep 26, 2025
eudelins-zama
force-pushed
the
feat/keygen_via_gateway
branch
from
801d1f7 to
c269808
Compare
isaacdecoded
changed the title
isaacdecoded
force-pushed
the
feat/keygen_via_gateway
branch
from
c4bc347 to
cdcecd9
Compare
isaacdecoded
force-pushed
the
isaac/383/test/fhe-keygen-through-gateway
branch
from
631ca9a to
353918c
Compare
isaacdecoded
commented
Sep 30, 2025
isaacdecoded
changed the title
* feat(gateway-contracts): implement single keygen * test(gateway-contracts): fix KmsManagement tests and few contract logic * fix(gateway-contracts): adjust Decryption event param names and fix mock tests/contracts * chore(gateway-contracts): rollback to keygen method name * feat(gateway-contracts): introduce keygen threshold and clean KmsManagement from deprecated vars * refactor(gateway-contracts): rename KmsManagement contract to KMSManagement * chore(gateway-contracts): update bindings and mocks * chore(gateway-contracts): execute git mv for case-sensitiveness * chore(gateway-contracts): execute git mv for case-sensitiveness * fix(gateway-contracts): include missed keygen threshold parameter on upgrade * refactor(gateway-contracts): rename task file for FHE keys gen --------- Co-authored-by: Isaac Herrera <isaac.herrera@zama.ai>
…ock tests/contracts
…gement from deprecated vars
* feat(gateway-contracts): implement single keygen * test(gateway-contracts): fix KmsManagement tests and few contract logic * fix(gateway-contracts): adjust Decryption event param names and fix mock tests/contracts * chore(gateway-contracts): rollback to keygen method name * feat(gateway-contracts): introduce keygen threshold and clean KmsManagement from deprecated vars * refactor(gateway-contracts): rename KmsManagement contract to KMSManagement * chore(gateway-contracts): update bindings and mocks * chore(gateway-contracts): execute git mv for case-sensitiveness * chore(gateway-contracts): execute git mv for case-sensitiveness * fix(gateway-contracts): include missed keygen threshold parameter on upgrade * refactor(gateway-contracts): rename task file for FHE keys gen --------- Co-authored-by: Isaac Herrera <isaac.herrera@zama.ai>
* feat(gateway-contracts): implement single keygen * test(gateway-contracts): fix KmsManagement tests and few contract logic * fix(gateway-contracts): adjust Decryption event param names and fix mock tests/contracts * chore(gateway-contracts): rollback to keygen method name * feat(gateway-contracts): introduce keygen threshold and clean KmsManagement from deprecated vars * refactor(gateway-contracts): rename KmsManagement contract to KMSManagement * chore(gateway-contracts): update bindings and mocks * chore(gateway-contracts): execute git mv for case-sensitiveness * chore(gateway-contracts): execute git mv for case-sensitiveness * fix(gateway-contracts): include missed keygen threshold parameter on upgrade * refactor(gateway-contracts): rename task file for FHE keys gen --------- Co-authored-by: Isaac Herrera <isaac.herrera@zama.ai>
* feat(gateway-contracts): implement single keygen * test(gateway-contracts): fix KmsManagement tests and few contract logic * fix(gateway-contracts): adjust Decryption event param names and fix mock tests/contracts * chore(gateway-contracts): rollback to keygen method name * feat(gateway-contracts): introduce keygen threshold and clean KmsManagement from deprecated vars * refactor(gateway-contracts): rename KmsManagement contract to KMSManagement * chore(gateway-contracts): update bindings and mocks * chore(gateway-contracts): execute git mv for case-sensitiveness * chore(gateway-contracts): execute git mv for case-sensitiveness * fix(gateway-contracts): include missed keygen threshold parameter on upgrade * refactor(gateway-contracts): rename task file for FHE keys gen --------- Co-authored-by: Isaac Herrera <isaac.herrera@zama.ai>
* fix(coprocessor): gw-listener, minor improvements * feat(coprocessor): gw-listener, ActivateKey, ActivateCrs * fix(coprocessor): tfhe-worker, bump tfhe-zk-pok to 0.7.2 To support test parameters for all operations * feat(coprocessor): sns-worker, accept missing keys * fix(test-suite): gateway keygen * fix(test-suite): disable flappy test * fix(coprocessor): wip, build non concurrent
isaacdecoded
force-pushed
the
isaac/383/test/fhe-keygen-through-gateway
branch
from
ed81c04 to
c8f0cc9
Compare
paulo-zama
pushed a commit
that referenced
this pull request
Mar 26, 2026
* chore: update alloy deps * chore: bump chainguard rust images to 1.93
mergify bot
pushed a commit
that referenced
this pull request
Mar 26, 2026
* fix(relayer): exclude extra data from serialization - See comments for explanation. * chore(relayer): pg_cron moved to app cron * chore(relayer): applying changes for cron * chore(relayer): adding debug logs for cron chore(relayer): removing cron from ci * chore(relayer): moving cron tasks chore(ci): increasing pg max connections for test ci chore(relayer): applying changes for err_reason chore(ci): increasing max connection for ci chore(ci): increasing ulimit of rust tests * chore(relayer): adding default config for test suite to skip intensive cron * chore(relayer): adding config for expiry and cron interval - still to apply: durations * chore(relayer): passing sql interval settings in todo * chore(relayer): removing y settings support for intervals * fix(relayer): handle the ext_job_id returned from db correctly Correctly handle ext_job_id from db: - Add tests to ensure consecutive duplicate requests are failing in case of v2 (since an unassigned ext_job_id is returned to the user). v1 does not have this issue, as the ext_job_id is not used. - The tests only assert if job ids are valid by making GET query. - Then fix the bug by handling the ext_job_id returned from db correctly and then returning it to the user for user decrypt and public decrypt. UNIQUE constraint in DB: - Previously the ext_job_id was unique only when status is not failed or timed out. So, when a new request has to be made, it can have same ext_job_id. But we want it to always be unique. - So, add missing UNIQUE constraints on ext_job_id to enforce the uniqueness at DB level. * fix: handle orchestrator dispatch errors in V1 handlers * fix(relayer): atomic transaction for share insertion and completion - Previosly, doing these two updates in separate transaction could lead to race condition where a request can be timed out by the cron between these two transactions. - To remove this possibility, ensure both the operations are done atomically in a single transaction. * feat(relayer): update models for consistency - Add GatewayBlockNumber model, to make this repo consistent with others and update the query to use it. - Include tx hash in UserDecryptShare model. Also, create a model for user decrypt share since it has many parameters. * chore(relayer)!: move retry-after to header and add tests - Previously, the retry after was in body for 202 responses. - Now, to align 429 response (which returns the retry-after in header) move it to header and hence remove the field in response. - Also, update tests to assert this. * refactor(relayer): rename the ext_job_id from two sources for clarity * chore(relayer): update version in cargo.toml -> v0.8.0-rc.5 * chore(relayer): wip metrics * chore(relayer): metrics input proof status * chore(relayer): adding status metrics update to pub dec * chore(relayer): metrics wip adding histoVec for status timing * chore(relayer): adding updated at metrics for statuses pub dec * chore(relayer): adding metrics for statuses for user dec * chore(relayer): alerting on statuses document to be refined chore(relayer): metrics doc for statuses * chore(relayer): metrics on active and idle connection and acquire chore(relayer): metrics on active and idle connection and acquire * chore(relayer): metrics for query durations and failures * chore(relayer): sql metrics documentation * chore(relayer): versionning http metrics chore(relayer): metrics documentation for http legacy and v2 * chore(relayer): transaction metrics declaration * chore(relayer): clippy sat sub * chore(relayer): metrics for v2 keyurl * fix(relayer): fix metrics init by adding health handler * chore(relayer): Adding metrics to share reception queries * chore(relayer): applying metrics to block repo * chore(relayer): adding metrics to in app cron timeout * chore(relayer): adding metrics to expiry cron * chore(relayer): clippy * chore(relayer): integrating engine metrics * chore(relayer): bump v0.8.0-rc.6 for metrics * chore(relayer): applying changes refactor to metrics * chore(relayer): adding configurable bucket for transaction * test(relayer): fix keyurl test * ci(relayer): add missing key url test run in ci * ci(relayer): in ci tests, use add timeout and use make cmds - Using make commands ensures we can use identical commands for testing in local and ci. * chore(relayer): make two conn pools, one each for app & cron - Update health checks and monitor metrics to monitor both pools. * chore(relayer): attempt once to init sql pool and report failure - Previously, the pool was trying in infite loop to aquire connections. - Now, reports failure after first attempt, if not able to acquire. * chore(relayer): use task manager to spawn cron & monitor task - Previously, these tasks were spawned in orphaned handles and not receiving the shutdown signal. - Now, integratem them into orchestrator's task manager. So that the tasks shutdown properly. * fix(relayer): min connections to 0 in tests * test(relayer): add proper shutdown for relayer in tests - Previously Drop-based cleanup only signaled cancellation without waiting for completion, causing PostgreSQL connections to accumulate across tests and hitting the 100 connection limit in CI. - Add async shutdown() method that waits for relayer handle completion ensuring proper connection cleanup before test finishes. * chore(relayer): fix internal server error message * chore(relayer): updating to v0.8.0-rc.7 * chore(relayer): improve logging in gw listener - Log block_number, block_hash, log_index logging, topic0 and topic1 for better debugging. * feat(relayer): add parallel listeners with deduplication - Update block number repo to include instance id. This is an integer assigned at startup (0, 1, 2 and so on.). Will be used to update the corresponding row in block number store. - Add a moka based deduplicator that is used to deduplicate events from multiple listeners and update in the DB only once. It also has TTL based automatic cleanup. - Spawn multiple listenrs during startup and de-duplicate before updating to DB. * refactor(relayer): refactor listener into modular functions - As a preparation for app level restart of listener. * chore(relayer): move listener reconnection logic from lib to app - Previously, listener reconnection was implicitly handled by the library, which was not visible / transparent at the app level. - Now, disable library level reconnection and do it in app leve. * chore(relayer): add in-flight status in migrations - This is in preparation to adding long queueing support for requests. Updating the migrations, so that we don't need to do another change in near future. * chore(relayer): bump version to v0.8.0-rc.8 * chore(relayer): bump version to v0.8.0 * test(relayer): add missing tests for parallel listeners - Previously, we added parallel listeners feature, but the functional tests for this were missing. - To enable these tests, extend the mock to support emitting events to specific subscriptions (based on indices: first, second, third etc.,). Adapt fhevm wrappers as well to the updated mock API. - Add new test cases to simulate different scenarios of missed events. Eg: missed by none, missed by 1/2, 1/3 or 2/3. Missed by 3/3 will lead to timeout, which is covered in timeout test. - Also, include targets for mock library and long running tests in makefile. * test(relayer): make cron configurable and add tests - Make the cron values conifgurable and update the queries to use this values. Update default settings as well. - Add tests for response timed out case by disabling mock mode and setting low values for timeout. * test(relayer): add negative test cases for transaction errors * chore(relayer): separate preparing transaction and transaction loop chore(relayer): keep full flow func * chore(relayer): queuing wip * chore(relayer): requests for tx in flight status update * chore(relayer): hooks and in flight status integration chore(relayer): removing pool from first iteration * chore(relayer): clippy * chore(relayer): passing receipt extraction to hook for input proof removing useless imports * chore(relayer): removing helper returning receipt * chore(relayer): remove send raw tx sync from engine, separated * chore(relayer): upgrading ruint and alloy to address cve * fix(relayer): add request-id and log (warn) rate limit errors * chore(relayer): merge 504 errors into 503 errors - As documented in comments, 504 errors are overidden by Cloudflare and this cannot be disabled in our setup. So, we use 503 instead while keeping the distinct labels. * chore(relayer): mempool struct * chore(relayer): queue, wip before error dispatcher * chore(relayer): queue integration and error dispatch for input proof chore(relayer): clippy * chore(relayer): adding queuing to public decrypt * chore(relayer): queue integration in user decrypt flow * chore(relayer): adding tx queue configuration * chore(relayer): adding throttling queue to orchestrator and renaming * chore(relayer): changing unbounded to bounded - Adding new EventProtocolError -> protocol overwhelmed. - Settings for queue capacity and tps with validator > 0 - senders panic following die-fast principle, if channel is closed, should not happens. - Create new failure mode when queue is full. - WIP: 503 with protocol_overwhelmed label * chore(relayer): adding error management to queue and refactor * chore(relayer): soft_capacity added to throttler * chore(relayer): moving throttler instanciation * chore(relayer): bouncing applied on public decrypt v1 v2 * chore(relayer): adding bouncing onto user_decrypt * chore(relayer): bouncing on input proof * chore(relayer): update default config values for queue * refactor(relayer): fix a naming issue - This is indeed a (generic) gateway handler and not key url specific gateway handler. * chore(relayer)!: use rate_limited error label for protocol overloaded - See the comments in code for explanation. * feat(relayer): add admin endpoint for updating throttle rate - The endpoint allows to update the throttler rate while the application is running. - It can be enabled or disabled in config. - Could be enabled to run in Testnet to fine tune the throttle rate without restarting / redeploying the relayer. - Could be disabled in production via config, if deemed unsafe. * refactor(relayer): use derive default for UsageLimit enum * chore(relayer): release v0.8.2 * chore(relayer): settings for readiness check * chore(relayer): wip on public decrypt readiness queue * chore(relayer): public decrypt readiness flow * chore(relayer): pub dec readiness bouncer * chore(relayer): user decrypt readiness processor * chore(relayer): readiness check proc user decrypt * chore(relayer): integration fhevm readiness user dec * chore(relayer): user decrypt readiness bouncing * chore(relayer): clippy * chore: minor dockerfile improvements * chore(relayer): declaring one tx queue per flow * chore(relayer): throttlers declaration refactor * chore(relayer): queue metrics declaration * chore(relayer): adding metrics to queues * chore(relayer): init metrics for throttlers unit tests * chore(relayer): rename fhevm_relayer.rs to startup.rs for clarity - Rename to reflect the purpose of this file. Update doc string to document the start functuion. * feat(relayer): add startup recovery - Fetch queued, processing and tx_inflight requests and re-dispatch them. * chore(relayer): reset in-flight->processing during recovery * chore(relayer): add tests for startup recovery * fix(relayer): deserialize ct handles to fix recovery * chore(relayer): treat extra data as string everywhere internally - extra data is given as string in http api and converted from bytes to string when received as part of blockchain event. So, remove unnecessary intermediate conversions to Bytes. * chore(relayer): make repo updates idempotent - Use more strict condition in WHERE claude to make all DB updates idempotent. * fix(relayer): handle user decrypt insert share response correctly - Previously, when a request share insert failed, there were different possibilities and all of them were handled in the same way. This lead to errors, when database updates were made idempotent. - Now, differentiate the scenarios and handle them properly. * feat(relayer): add configurable cron startup delay after recovery - After restart, there a small interval is needed for listener to catch up on events from blocks produced while relayer was down and update the responses accordingly, before the requests are timed out by timeout cron. - To accomodate this, the timeout cron and expiry cron are started with a configurable delay. The delay has a rule of not more than 10% of timeout or expirty interval to ensure, the request's timeout / expirty is still within a 10% error margin (during restarts). - Also, set this delay to zero in tests that are testing timeout. Otherwise the timeout crons don't run within the test's run period. * feat(relayer): handle reverts due to pause, funds insuff, invalid signs - Previsouly, these were returned as generic transaction errors. - Now, they are returned as known errors with specific labels. * fix(relayer)!: make a new ext_job_id when repeating failed/timed_out request - Previously, when an int_job_id in decrypt (user/public) failed and a new attempt was made, the already existing job was reactivated in sql and ext_job_id was reused. - Now, a new entry is created and the new ext_job_id is returned to user. - Update status update queries to capture metrics correctly as per the new logic. - Also, add a test to verify this. * fix(relayer): remove 408 response on v1 pub decrypt - Bug: Only in public decrypt endpoint v1, if response (success or error) is not received in 4s, then a 408 response was sent. - Fix: Remove this logic. If it takes too log, then it will timeout. * fix(relayer): add deduplication at http handler for v1 pub decrypt - Previously, requests were not deduplicated at http handler level, so multiple events for request processing were dispatched, leading to duplicate active requests in the queue and hence uninteded behavior as well. Just to make the test pass, this was covered by having a timeout branch, but ths was a bug introduced. - Now, remove the timeout branch and deduplicate at http handler properly without race conditions to avoid duplicate active requests entering the system (queued, processing). * chore(relayer): extend deduplication to other three cases as well * chore(relayer): use same conn for fetching shares * test(relayer): per test schema in same db for isolation * ci(relayer): install psql natively for more conn * fix(relayer): type in throttler name via /admin * chore(relayer): version bump * chore(relayer): fix clippy errors * feat(relayer): make each listener url configurable - So that, they could be pointed to different nodes for not having single point of failure. * chore(relayer): filter by event topics in ws subscription * chore(relayer): recycle ws connections on listener * feat(relayer): add polling based listener - Add support for polling based listener, that can be used to be resilient against missed events in web sockets. * chore(relayer): continue after max retries for ws reconn - For websocket reconnect, don't return but continue after max retries while logging at error level instead of warn. * fix(relayer): do nothing on dupl event in gw handler - Previously, when the idempotent query updating the database returned zero affected rows, it was not differentiated into possible reasons (for input proor and public decrypt). So, this led to generic handling for all scenarios. - Now, differentiate them in the repo functions and handle them properly in the gateway listeners. * fix(relayer): redact private key in logs * chore(relayer): bump to v0.8.5 * chore(relayer): adding array to yaml seq conf * chore(relayer): bumping version to relayer v0.8.6 * chore(relayer): separate config for read http url * fix(relayer): race condition in user decrypt queries - duplicate detection had two separate queries allowing shares to be deleted between them, now wrapped in single transaction - ON CONFLICT refreshed timestamp causing request and shares to diverge on expiry, now keeps original timestamp like public_decrypt - expiry deleted shares independently of parent request allowing orphaned states, now uses JOIN to delete only when parent also expires * chore(relayer): handle revert acl selector - Handling ACL selectors errors retries for node state inconsistency. - retrying max_retries - 1 to not pass in max_retry_exceeded error mode. - introducing a specific metric. * feat(relayer): dynamic retry-after based on queue status - Compute Retry-After header dynamically using queue size and drain rate instead of static values, so clients poll at adaptive intervals - Split computation into type-specific methods: input proof uses single TX queue, decrypt operations use dual queue (readiness + TX) - Also extend admin config for runtime-tunable parameters (processing times, safety margin, backoff intervals) * fix(relayer): address clippy warnings in recovery tests - Add #[allow(dead_code)] to unused test helpers - Replace &[handle.clone()] with slice::from_ref(&handle) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(relayer): show JobId as hex in logs * fix(relayer): decrement metrics on expiry purge - Previously, the expired request count metrics was not decrementing relayer_request_count when deleting expired records, causing metrics to drift from actual DB state. * fix(relayer): init status metrics from DB on startup - Previously, metrics gauges were initialized to 0 on startup, causing negative values when existing requests completed or were purged. - Now queries DB counts before any recovery operations. * chore(relayer): redact uneeded data from logs * fix(relayer): redact blockchain RPC URLs in logs - Also, switch to Derivative for redacting private key. * fix(relayer): pass full error upstream - e is already GatewayTxnError, so removing the wrapping. * chore(relayer): handle conversion errors * chore(relayer): alert log on fail w/ empty reason * docs(relayer): add logging policy docs * feat(relayer): add logging primitives * chore(relayer): improve logs in tx engine, helper * chore(relayer): improve logs in tx throttler * chore(relayer): improve logs in tx processor * chore(relayer): improve logs in cron workers * chore(relayer): improve logs in cron workers * chore(relayer): improve logs in public decr handlers * chore(relayer): improve logs in user decr handlers * chore(relayer): improve logs in input proof handlers * chore(relayer): bump relayer & migrate to v0.8.7 * fix(relayer): clippy issue (use fallible incorrectly) * feat(relayer): dedup + job based for input proofs - Previously, each input proof request, even if it has the same content, triggered separate jobs on Gateway. This was needed, because they will get same results. - Now, add dedup for input proof requests like for decryptions. Uses ON CONFLICT with partial unique index for atomic deduplication, avoiding race conditions from check-then-insert. - In migrations, set the content hash value for existing requests to the zero value. - Also, simplify job id by handling only content hash variant instead of two (as we don't need UUID V7 for job tracking in any flows anymore). - More context: Earlier there was a misunderstanding that each input proof request (even for same content) could result in different handles. This is incorrect. Its only applicable when generating the ZKPoK itself on the client side that, when for same plain text inputs, different ZKPoKs will be generated when repeated. But for a given ZKPoK + other parameters, result from copro will always be the same. * chore(relayer): improve logs in input proof handlers * fix(relayer): improved logs + config validation * chore(relayer): down migration for v0.8.8 -> 0.8.7 * chore(relayer): bump relayer & migrate to v0.8.8 * chore(relayer): revise eta computation formulae * feat(relayer): incldue ecdsa errors to inv sign * chore(relayer): bump relayer & migrate to v0.8.9 * fix(relayer): correct return type for inv sig * chore(relayer): bump relayer & migrate to v0.8.10 * fix(relayer): make clippy happy (#850) * fix(relayer): update tests for invalid_signature API change (#851) * fix(relayer): update tests for invalid_signature API change Updates tests that were not adapted after commit 34a94ad which changed the invalid_signature error response format from label "invalid_signature" to label "validation_failed" with a details array. Test fixes: - input_proof_v2_test.rs: expect validation_failed + signature details - public_decrypt_v2_test.rs: expect validation_failed + signature details - user_decrypt_v2_test.rs: expect validation_failed + signature details - transaction_helper_test.rs: add missing JobId argument to method calls - schema_isolation_test.rs: use sqlx::query().bind() instead of query! macro - recovery_test.rs: remove duplicate std::slice import * refactor(relayer): simplify signature details assertion * refactor(relayer): check full error JSON in invalid_signature tests * revert(relayer): restore sqlx::query! macro in schema_isolation_test * fix(relayer): restore .sqlx query cache * fix(relayer): update sqlx cache with test queries * fix(relayer): include all targets in sqlx-prepare * fix(ci): use bracket notation for jq service map access (#852) jq interprets $service_map.relayer-migrate as subtraction (relayer minus migrate function call), causing "migrate/0 is not defined" error. Use bracket notation for all keys for consistency and to prevent similar issues with future hyphenated service names. * fix(relayer): correct read_http_url validation and error message (#853) * fix(relayer): correct read_http_url validation and error message - Fix validation to check for "http://" prefix instead of just "http" - Fix error message to display read_http_url instead of http_url * chore: fmt * feat: implement delegated user decryption use case for V2 endpoints * feat: implement delegated user decryption use case for V2 endpoints * refactor: check for user decryption readiness on delegated user decryption proc * chore: fix missed not intended changes * feat: separate settings and implement bouncer for delegated user decryption readiness * feat: implement user decrypt request type modeling * chore: apply format * chore(relayer): bump relayer & migrate to v0.9.0-rc.2 * fix(ci): upgrade raven-actions/actionlint to v2.1.1 - Issue: v2.0.0 fails with ERR_PACKAGE_PATH_NOT_EXPORTED on Node.js 22 due to unpinned `npm install @actions/tool-cache` pulling a version with broken ESM exports - Fix: v2.1.1 pins @actions/tool-cache@3.0.1 and uses actions/github-script@v8.0.0 - Ref: https://github.com/raven-actions/actionlint/releases/tag/v2.1.1 * fix(relayer): make updated_at trigger conditional on actual row changes - Previously, the trigger updated the value event during cache reads in POST handler where we used ON CONFLICT (a dummy update) to read detect conflict and read without race conditions in a single query. - Now, the trigger is updated to change the timestamp only the value of the row has distinctively changed. * fix(relayer): skip timestamp update in input proof ON CONFLICT - Previously, the input proof ON CONFLICT clause updated the timestampt to NOW(), which was a bug. It should actually be unmodified. - Now, set it to the same value. * fix(relayer): add status field to V2 POST error responses - Previously, V2 POST handlers returned AppResponse for early-stage errors, which produced {"error": {...}} format without a top-level status field. This was inconsistent with V2 responses format. - Now, V2 POST error responses use RelayerV2ResponseFailed which produces {"status": "failed", "request_id": "...", "error": {...}} format. - Also, update RelayerV2ResponseFailed helpers to convert field names to camel case in error details. * chore(relayer): use enum for response.status field * test(relayer): update integ test to validate status * test(relayer): add negative test cases in v2 tests - For now, just duplicat the negative test cases from v1 to v2. - Later, refactor to avoid duplication. * fix(relayer): use UUIDv4 for request_id in HTTP handlers - Previously, the internal_request_id (UUID v7) was mistakenly used in HTTP handlers. This was intended only for older version (now not there) of input proof requests, where each external request maps to a new internal requests. Now, each requests maps to exisitng or new job. - Now, use UUID_v4 in all places. * feat(relayer): add HTTP response logging for V2 endpoints Add structured logging for all V2 HTTP responses with consistent fields: - 200/202 success: request_id, http_status, ext_job_id - 4xx errors: request_id, http_status, label, message - 5xx errors: request_id, http_status, label, message All logs use info level with "HTTP response" message for easy filtering. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * feat(relayer): add HTTP response logging for delegated user decrypt Add structured logging for delegated user decrypt POST endpoint. Also fixes status field to use ApiResponseStatus::Queued enum instead of hardcoded string for consistency. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(relayer): race due to send_raw_tx_sync latency spike - Sometimes, there can be a spike in send_raw_transaction_sync API to as high as 1.5s and in these cases, the event can actually arrive much before the call completes, resulting in a race condition. - Fix this temporarily by adding a retry logic in gateway handlers while processing the response. - Also, group the common config in user decrypt handlers to avoid too many arguments issue from clippy. * fix(relayer): port V2 response fixes to delegated user decrypt handler Apply missing changes from earlier commits to delegated user decrypt: - Use Uuid::new_v4() for request_id instead of internal_request_id - Use RelayerV2ResponseFailed for error responses instead of AppResponse to ensure consistent V2 response format with status field Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix(relayer): fix case conversion bug for err msg * fix(relayer): reorder migrations due to hot fix - Some changes from release v0.8.x branch were cherry picked and included a DB migration that is already applied on the deployment. This branch also had some new migration (related to user decrypt). - Since migrations must be in the order they'll be applied, reorder the user decrypt migration after the migrations in the hot fix cherry picks. * fix(relayer): dependabot issue by upgrading package * fix(relayer): add missing error handlers for deleg - Delegated user decrypt was missing handling of some error paths, fix it. - Add comprehensive tests for delegated user decrypt covering all positive and negative paths. * chore(relayer): bump relayer & migrate to v0.9.0-rc.3 * chore(relayer): remove Tower rate limiter middleware (#865) * chore: remove tower rate limiter * chore: run sqlx-prepare * chore(deps): update time crate * refactor: simplify post routes merge * chore: remove code outside relayer scope (#869) * chore: remove deprecated apps (back, front, oracle, portal) * chore: remove non-relayer Docker configs and compose files * chore: remove deprecated scripts and tech-spec docs * chore: remove fhevm git submodule * chore: remove pnpm workspace files * chore: clean up CI workflows for relayer-only repo * chore: clean up root configs for relayer-only repo * docs: rewrite README for relayer-only repository * chore: remove forge-std submodule * chore(relayer): add test coverage tooling (#953) - integrate cargo-llvm-cov - generate HTML coverage report - add CLI coverage output * chore: update README (#870) * docs: update README * docs: update wallet setup * refactor: reword introduction * chore: add ToC * refactor: clarify relayer config subsection * refactor: showcase pre-built template * refactor: add details on wallet setup * chore: fix public decryption intro * chore: add note on Testnet $ZAMA * chore: update alloy deps (#876) * chore: update alloy deps * chore: bump chainguard rust images to 1.93 * ci: simplify CI for relayer-only repository (#873) * ci: refactor ci * chore: update codeql-action version * chore: update checkout version * fix: fix zizmor template injection risk * chore: update zgosalvez/github-actions-ensure-sha-pinned-actions * chore: update cargo-bins/cargo-binstall * chore: update docker actions * chore: update slsa-framework and actions/download-artifact * chore: update dtolnay/rust-toolchain and Swatinem/rust-cache versions * chore: remove unused BOT_USERNAME secret * ci: revert dtolnay action version upgrade * ci: make rust test postgres setup runner-portable * ci: skip docker build job when service matrix is empty * chore: fix lint * ci: use chainguard postgres image for postgres * chore(relayer): relayer-sdk metrics (#986) (#872) - extract from headers relayer-sdk data - relayer-sdk-name - relayer-sdk-version * feat: enhance makefile devx (#877) * feat: enhance makefile devX * docs: update CLAUDE.md and move to root * chore: update keccak to v0.1.6 * refactor: refactor devnet/testnet preflight target * ci: skip _check-postgres guard as postgres running in service container * chore: remove docker-push target * feat: add allowance target * docs: refactor README to use makefile targets (#880) * docs: refactor README to use makefile targets * fix: fix list nesting * refactor: reword E2E test setup instructions for devnet * chore(relayer): documentation update - deletion policy (#1074) * chore: remove unused deps, update deps (#878) * chore: remove unused relayer deps * chore: remove unused ethereum_rpc_mock deps * chore: update relayer-migrate deps * chore: update ethereum_rpc_mock deps * chore: update relayer deps * chore: cargo update * chore: fmt * fix(relayer): no matching rows on SQL insert/update (#989) - Change deletion order from user_decrypt/share to share/user_decrypt - Add additional logs for improved troubleshooting * chore(relayer): Cleanup custom metrics (#951) - Unnecessary metrics were removed - SQL monitoring should be handled at database level * test(relayer): prove queued-to-failure bug Add 3 integration tests (one per request type) that demonstrate requests stuck in 'queued' status when readiness checks fail with a contract error. Before the handler fix, these tests panic with "Request did not reach terminal state in time" because update_status_to_failure_on_tx_failed silently no-ops on queued requests. Also adds set_readiness_contract_error() mock helper, set_readiness_success_after_n_failures() for staged readiness, parameterized create_readiness_config() with a new new_with_minimal_readiness() constructor (2x50ms), and SQL functions update_status_to_failure_from_queued with public wrappers for readiness_check_failed and enqueue_failed. * fix(relayer): queued-to-failure SQL transitions Update handle_error call sites to use correct SQL functions: - ReadinessCheckFailed uses update_status_to_failure_on_readiness_check_failed - ReqRcvdFromUser uses update_status_to_failure_on_enqueue_failed These use WHERE req_status = 'queued' instead of 'processing'/'tx_in_flight', allowing the transition to match. * test(relayer): assert error labels in V2 tests - Previously: V2 error tests checked HTTP status and response status but not the error label field - Now: every tested error path asserts its label. - Also, add a test for readiness check timed out label. * fix(relayer): fix timeout label assigning in V2 GET - Issue: all TimedOut requests returned "response_timed_out" even when the timeout was a readiness check - Fix: check err_reason against READINESS_CHECK_TIMEOUT_MSG to return "readiness_check_timed_out" when appropriate (public_decrypt and user_decrypt only — input_proof has no readiness checks) * fix(relayer): use correct selector for delegated decrypt - Issue: delegated decrypt mocks used userDecryptionRequestCall selector instead of delegatedUserDecryptionRequestCall — requests never reached terminal state - Fix: parameterize the TX selector per decrypt kind; as part of this, unify 8 separate direct/delegated mock methods into 5 via a new UserDecryptKind enum - Removed unused _result: Bytes param from success methods * ci(relayer): add delegated user decrypt test to ci - Makefile command is used to run tests in CI and was missing delegated user decrypt tests. * chore(relayer): improve comments * test(relayer): rm redundant test in test utility - If the setup of wrappers panic, its the tests in relayer/tests that will panic. It will already provide enough information to identify and debug if any issue in the test util. - So, this test for test util seems redundant. * fix(relayer): v2 proof rejection returns 200 OK - Issue: V2 GET /input-proof/{job_id} returned 500 when the coprocessor rejected a proof, because classify_error() could not extract an ABI selector from the plain-text "Proof Rejected" string and fell through to RevertReason::Unknown - Fix: handle the proof-rejection path (Completed + accepted=false) before classify_error(), returning 200 OK with accepted=false; classify_error() now only applies to the Failure path (actual TX reverts) Fixes zama-ai/fhevm-internal#1120 * refactor(relayer): deduplicate classify_error - Previously: classify_error was duplicated across 3 V2 handlers with a generic name that invited misuse for non-revert errors - Now: single classify_revert_error in v2/types/error.rs, called from all handlers; name makes the ABI-selector-only scope clear Refs zama-ai/fhevm-internal#1109 * feat(relayer): validate ACL on host chain directly With the simple ACL feature, the multichain ACL contract is removed from the gateway. ACL checks must now be performed directly on each host chain. - Add HostAclChecker that multicalls ACL.isAllowed on host chain RPCs - Extract chain ID from ciphertext handles to route to correct chain - Move HostChainIdChecker to http layer for early unsupported-chain rejection with a descriptive error label - Consolidate BounceChecker fields into a struct (motivated by growing constructor parameters) - Use Display formatting for ACL failure messages * feat(relayer): remove MultichainACL bindings - Bump fhevm_gateway_bindings to fhevm commit b27c89f from feat/1094/remove-multichain-acl-v0.12 branch. ACL validation now happens directly on the host chain instead of via a separate MultichainACL contract on the gateway. - Remove all MultichainACL and MultichainACLChecks error handling from the relayer since these contracts no longer exist in the gateway. - Update isUserDecryptionReady call to match the new contract signature which no longer takes a user_address argument. * test(relayer): add host ACL integration tests - Add ACL allow/deny/RPC-error scenarios for public decrypt, user decrypt, and delegated user decrypt v2 endpoints - Add on_call_dynamic to mock server for runtime-configurable responses - Embed valid chain ID in random_handle() so HostChainIdChecker passes in all existing tests - Add multi-handle and cross-chain ACL test cases * fix(relayer): redact RPC URLs from contract errors - Issue: alloy::contract::Error wraps reqwest::Error whose Display appends "for url (https://...)", leaking RPC endpoints (possibly with embedded API keys) into logs, DB err_reason, and HTTP responses - Fix: add redact_alloy_error() helper that walks the error source chain, finds the reqwest::Error, and strips the URL suffix using reqwest::Error::url() * refactor(relayer): split host/gateway/readiness - Previously host-chain logic (ACL checker, chain ID validation) and gateway ciphertext checks were mixed under gateway/readiness_check/ - Now each module has a single responsibility: host/ for host-chain interactions, gateway/ for gateway-chain interactions, readiness/ composes both as a cross-chain orchestrator * chore: remove v1 routes (#891) * chore(relayer): delete V1 endpoint handlers and types * chore(relayer): remove V1 handler wiring from HTTP server * chore(relayer): remove OnceHandler infrastructure * chore(relayer): migrate V1 type references to V2 * chore(relayer): delete V1 integration tests * chore(relayer): remove V1 smoke test targets from Makefile * fix(relayer): remove V1 route from keyurl integration test * chore(relayer): remove stale V1 OpenAPI specs * fix(relayer): add keyurl_v2 to OpenAPI paths * chore(relayer): remove dead HttpApiVersion::V1 enum variant * chore(relayer): remove stale V1 references from comments * chore(relayer): remove redundant smoke test target * docs(relayer): update README to reflect V2-only API * chore: cargo fmt * docs(relayer): remove stale V1 references from docs * refactor: move event_id mapping to inner enums (#894) * fix(relayer): correct typo suscribers -> subscribers in event dispatcher * refactor(relayer): delegate event_id() to inner enums in event.rs * chore: update deps quinn-proto * refactor: remove orchestrator deadcode generics (#895) * refactor(relayer): remove dead hook infrastructure from Orchestrator * refactor(relayer): replace generic Orchestrator with concrete type * docs: update CLAUDE.md to reflect Orchestrator refactor * chore: fmt * refactor: remove EventDispatcher/HandlerRegistry traits and make TokioEventDispatcher concrete * chore: fmt * fix(relayer): delete transaction_helper_test.rs which silently swallows errors (#897) * fix: move all validation to settings (#898) * fix(relayer): replace panic with Err for empty histogram buckets * fix(relayer): move address and listener pool validation into Settings::new() * test(relayer): add unit test for invalid address rejection in Settings::new() * refactor(relayer): narrow validate_addresses and validate_listener_pool_config to private * chore(relayer): update histogram bucket config - local.devnet.yaml.example - local.testnet.yaml.example - local.yaml.example * refactor: reduce throttler boilerplate (#900) * refactor(relayer): extract throttler factory functions to reduce init boilerplate * refactor(relayer): remove trivial create_readiness_throttler wrapper * refactor(relayer): remove create_tx_throttler wrapper, use local aliases instead * refactor: remove eyre for anyhow (#899) * refactor: replace eyre with anyhow in startup boundary * refactor: replace eyre with anyhow in gateway utils and console-mock * refactor: replace eyre with anyhow in test support * chore: remove eyre dependency from Cargo.toml * refactor: use .context() instead of .map_err(anyhow!) in parse_private_key * chore: fmt * docs: update CONTRIBUTING.md * chore(relayer): migrate once_cell::sync::OnceCell to std::sync::OnceLock (#902) * chore(relayer): ci - pipeline update - Clean up flows, jobs, and steps - Separate unit and integration tests - Align with fhevm CI standards - Remove deprecated flows, jobs, and steps - Bump quinn-proto version (from 0.11.13 to 0.11.14) * feat(relayer): add AWS KMS signer support (#903) - In our deployments, it is recommended to use AWS KMS Signer over private key signer for security concerns. Added support for this. - Private key and AWS KMS signing are selectable via explicit config (tx_engine.signer.type: private_key | aws_kms). - SignerConfig tagged enum enforces valid config at deserialization time; sensitive fields redacted in debug output - Also: replace two .unwrap() calls with ? propagation in TransactionEngine::new() (URL parse, private key parse) for graceful error handling * fix(relayer): removing filters from listener (#914) - Root cause: cannot unmarshal non-string into Go struct field EthSubscribeLogsFilter.topics of type common.Hash - Removing signatures from filters on subscribe logs for websocket streams - keeping addresses - keeping dedup cache - filter is already made on topics by the given handlers chore(relayer): compilation error * refactor: adjust extra data validations to be propagated verbatim * refactor: adjust extra data validations to be propagated verbatim * feat: propagate extraData on user decrypt response * feat: apply serde alias for extraData camel cased * fix: append 0x to user decryption response extra data field * refactor: rename alias by rename_all macro * fix(relayer): wiz security findings in Dockerfiles (#1165) - Pin package versions for reproducible builds - Add non-root USER instruction to postgres dev container - Ensure runtime stages run as non-privileged users (appuser) - Keep build stages as root (required for cargo cache mounts) * refactor: revise log levels (#920) * refactor(relayer): update logging taxonomy and policy for multi-relayer deployments * refactor(relayer): revise input proof handler logs for multi-relayer semantics * refactor(relayer): revise public decrypt handler logs for multi-relayer semantics * refactor(relayer): revise user decrypt handler logs for multi-relayer semantics * refactor(relayer): demote listener traffic noise to DEBUG for multi-relayer * fix(relayer): keep WebSocket recycle logs at INFO as listener lifecycle events * refactor(relayer): drop "local" from matched-request messages, clarify multi-relayer wording * fix(relayer): dependabot alerts and improve metrics (#1174) - Bump aws-lc version - Add a map to reduce cardinality - Update deny.toml (remove OpenSSL) * refactor: make data deletion opt-in with explicit `expiry_enabled` config flag (#921) * refactor(relayer): separate timeout and expiry background workers * docs(relayer): add section of timeout worker to README * refactor(relayer): add warn log when expiry worker is enabled * refactor(relayer): set timeout worker defaults to true * docs(relayer): update data retention policy on manual/automatic deletion * refactor(relayer): enforce timeout worker start * docs(relayer): refactor worker one-liner explainer * refactor(relayer): move expiry worker warn log to worker registration * refactor: add test-specific config file * fix: use old isUserDecryptionReady signature for backward compatibility (#925) * fix: reintroduce old signature isUserDecryption for backward compatibility * chore: update rustls-webpki * chore: remove default feature from aws-sdk-kms * chore: bump aws-config and aws-sdk-kms versions * refactor(relayer): unify V2 error response types (#915) - Previously: per-status error types (RelayerV2ApiError400, RelayerV2ApiError500, etc.) with serde_json::Value error fields in response structs - Now: single V2ErrorResponseBody enum with named constructors per error label, typed error fields in response structs * feat(relayer): update open api spec + cmd to gen it (#803) - Derive openapi.yml from utoipa annotations instead of hand-maintained YAML to eliminate spec drift; add openapi-export binary to serialize it - Centralize error labels in a declarative matrix that is auto-enriched into the spec with status codes, retry semantics, and SDK guidance - Add drift tests and CI lint step to catch mismatches on every PR - Serve Redoc UI at /docs for local browsing * docs(relayer): add mainnet self-hosting guide and restructure README (#916) * docs(relayer): add mainnet self-hosting guide and restructure README * chore(relayer): remove ci make target * chore(relayer): remove devnet ocurrences * docs(relayer): remove E2E Testnet section relying on Notion page * chore(relayer): use headings instead of emphasis for self hosting steps * chore(relayer): remove deprecated makefile target from readme * docs(relayer): specify gateway network in preflight balance check * docs(relayer): use bridge.zama.org url only * docs: sync timeout request with self-hosting guide * docs: remove internal faucet link * docs: add expiry worker config to self hosting guide * docs: add details to ProtocolPayment allowance check * fix: update mainnet config example * chore: remove duplicate expiry worker config reference entries * feat: add openapi-check to check target * chore(relayer): migrate relayer to fhevm (#914) - Remove unnecessary MD files - Remove .hadolint.yaml - Merge .gitignore contents * chore(relayer): migrate relayer to fhevm (#914) - Remove old .gitignore file * chore(relayer): migrate relayer to fhevm (#914) - Move files from apps/relayer to the root directory * chore(relayer): migrate relayer to fhevm (#914) - Remove CI before migration * chore(relayer): migrate relayer to fhevm (#914) - update scope * chore(relayer): migrate relayer to fhevm (#914) - fix typos --------- Co-authored-by: manoranjith <manoranjith.ponnuraj@zama.ai> Co-authored-by: Nicolas Boisde <nicolas.boisde@zama.ai> Co-authored-by: enitrat <msaug@protonmail.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com> Co-authored-by: Mathieu <60658558+enitrat@users.noreply.github.com> Co-authored-by: Isaac Herrera <isaacdecoded@gmail.com> Co-authored-by: malatrax <71888134+zmalatrax@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
E2E tests passing here: https://github.com/zama-ai/fhevm/actions/runs/18123599474
Closes https://github.com/zama-ai/fhevm-internal/issues/383