Add ciphertext drift e2e workflow

Author: Eikix

.github/workflows/test-suite-e2e-tests.yml

@@ -233,6 +233,11 @@ jobs:
         run: |
           ./fhevm-cli test hcu-block-cap
 
+      - name: Ciphertext drift test
+        working-directory: test-suite/fhevm
+        run: |
+          ./fhevm-cli test ciphertext-drift
+
       - name: Host listener poller test
         working-directory: test-suite/fhevm
         run: |

PLAN.md

@@ -0,0 +1,144 @@
+# Ciphertext Drift E2E Implementation Plan
+
+## Overview
+Add one focused end-to-end test path that proves the Rust `gw-listener` drift detector fires on an intentionally corrupted ciphertext digest in the existing e2e stack. Keep it compatible with the current `2-of-2` coprocessor workflow and reuse the existing Hardhat tests instead of building a new scenario from scratch.
+
+## Goals
+- Enable the Rust drift detector in the existing e2e coprocessor stack.
+- Add one deterministic drift-injection workflow that fits the current `fhevm-cli test` flow.
+- Verify detector output via metrics, not brittle log matching.
+
+## Non-Goals
+- No new general-purpose fault-injection framework.
+- No changes to the existing consensus watchdog behavior outside the dedicated drift test path.
+- No new CI profile or topology change beyond what the current workflow already deploys.
+
+## Assumptions and Constraints
+- Current workflow deploys `2` coprocessors with threshold `2`.
+- In `2-of-2`, one corrupted coprocessor means no consensus; this test should assert drift detection, not successful consensus.
+- The existing JS `consensusWatchdog` would intentionally fail this scenario, so the dedicated drift test run must disable it.
+- Host-side scripts can use `docker exec` against the shared Postgres container.
+
+## Requirements
+
+### Functional
+- `coprocessor-gw-listener` in e2e must receive `ciphertext_commits` and `gateway_config` addresses.
+- A script must mutate one ready-but-unsent `ciphertext_digest` row on a chosen coprocessor DB.
+- A dedicated `fhevm-cli test` target must orchestrate the injection and run an existing Hardhat test that produces ciphertext work.
+- The test must fail if the Rust `gw-listener` drift metric does not increment.
+
+### Non-Functional
+- Keep the change surface small and local to the existing e2e workflow.
+- Avoid adding new runtime services or test dependencies.
+- Keep the workflow deterministic enough for CI.
+
+## Technical Design
+
+### Data Model
+- No schema changes.
+- Reuse `ciphertext_digest(handle, ciphertext, ciphertext128, txn_is_sent, created_at)`.
+
+### Architecture
+- Extend the e2e `coprocessor-gw-listener` command with:
+  - `--ciphertext-commits-address`
+  - `--gateway-config-address`
+- Add a host-side drift injector script that:
+  - targets one coprocessor DB
+  - waits for a new ready row with `txn_is_sent = false`
+  - flips one byte in `ciphertext`
+- Add a host-side runner script that:
+  - stops one transaction sender
+  - starts the injector
+  - runs one existing Hardhat grep with the JS watchdog disabled for that process
+  - restarts the transaction sender
+  - checks `gw-listener` metrics for `coprocessor_gw_listener_drift_detected_counter`
+
+---
+
+## Implementation Plan
+
+### Serial Dependencies (Must Complete First)
+
+#### Phase 0: E2E Wiring
+**Prerequisite for:** Drift injection and runner script
+
+| Task | Description | Output |
+|------|-------------|--------|
+| 0.1 | Add detector address args to e2e `coprocessor-gw-listener` compose command | Drift detector enabled in e2e stack |
+| 0.2 | Add `GATEWAY_CONFIG_ADDRESS` to coprocessor env template used for multi-copro copies | Address available in all coprocessor env files |
+| 0.3 | Add a small plan file documenting the scoped implementation | `PLAN.md` |
+
+---
+
+### Parallel Workstreams
+
+#### Workstream A: Drift Injection
+**Dependencies:** Phase 0
+**Can parallelize with:** Workstream B
+
+| Task | Description | Output |
+|------|-------------|--------|
+| A.1 | Add a host-side script to wait for a new ready `ciphertext_digest` row in one coprocessor DB | `inject-coprocessor-drift.sh` |
+| A.2 | Mutate one byte in `ciphertext` for the selected handle and print the handle | Deterministic DB-level drift injection |
+
+#### Workstream B: Test Orchestration
+**Dependencies:** Phase 0
+**Can parallelize with:** Workstream A
+
+| Task | Description | Output |
+|------|-------------|--------|
+| B.1 | Add a host-side runner that pauses one transaction sender, launches the injector, runs one existing Hardhat test, and checks metrics | `run-ciphertext-drift-e2e.sh` |
+| B.2 | Add one `fhevm-cli test ciphertext-drift` entrypoint that calls the runner | Existing CLI can trigger the new path |
+
+---
+
+### Merge Phase
+
+#### Phase 1: Validation
+**Dependencies:** Workstreams A, B
+
+| Task | Description | Output |
+|------|-------------|--------|
+| 1.1 | Run targeted checks on the changed scripts and CLI wiring | Verified local changes |
+| 1.2 | Document how to run the new e2e path and what it proves | Clear operator/developer usage |
+
+---
+
+## Testing and Validation
+
+- Verify `coprocessor-gw-listener` command in e2e compose includes both new addresses.
+- Verify the injector script can discover and mutate a new unsent row for a chosen DB.
+- Verify the runner script disables the JS consensus watchdog only for this one intentional drift run.
+- Verify the runner fails if `coprocessor_gw_listener_drift_detected_counter` does not increase.
+
+## Rollout and Migration
+
+- No migration.
+- Hard cutover for the new e2e path: once merged, `fhevm-cli test ciphertext-drift` becomes the supported drift test entrypoint.
+
+## Verification Checklist
+
+- `bash -n test-suite/fhevm/scripts/inject-coprocessor-drift.sh`
+- `bash -n test-suite/fhevm/scripts/run-ciphertext-drift-e2e.sh`
+- `rg -n "ciphertext-commits-address|gateway-config-address" test-suite/fhevm/docker-compose/coprocessor-docker-compose.yml`
+- `rg -n "ciphertext-drift" test-suite/fhevm/fhevm-cli`
+
+## Risk Assessment
+
+| Risk | Likelihood | Impact | Mitigation |
+|------|------------|--------|------------|
+| The injector races before a ready row exists | Medium | Medium | Poll for a new ready row and gate on `txn_is_sent = false` |
+| The existing JS watchdog fails the intentional drift run | High | High | Disable it only for the dedicated drift test command |
+| Metric polling hits the wrong listener | Low | Medium | Check a specific gw-listener container and assert the exact counter name |
+
+## Open Questions
+
+- [ ] Whether we want a second scenario later for `3-of-5` where consensus still succeeds with one bad coprocessor.
+
+## Decision Log
+
+| Decision | Rationale | Alternatives Considered |
+|----------|-----------|------------------------|
+| Keep the first e2e on `2-of-2` | Matches current workflow with the least change | Adding a new CI topology now |
+| Use DB-level digest mutation | Smallest realistic fault injection point | Patching workers or faking on-chain events |
+| Assert via Rust metrics | More stable than log text matching | Parsing logs or relying only on the JS watchdog |

test-suite/fhevm/docker-compose/coprocessor-docker-compose.yml

@@ -111,6 +111,8 @@ services:
       - --verify-proof-req-database-channel=event_zkpok_new_work
       - --gw-url=${GATEWAY_WS_URL}
       - --input-verification-address=${INPUT_VERIFICATION_ADDRESS}
+      - --ciphertext-commits-address=${CIPHERTEXT_COMMITS_ADDRESS}
+      - --gateway-config-address=${GATEWAY_CONFIG_ADDRESS}
       - --kms-generation-address=${KMS_GENERATION_ADDRESS}
       - --error-sleep-initial-secs=1
       - --error-sleep-max-secs=10

test-suite/fhevm/env/staging/.env.coprocessor

@@ -43,4 +43,5 @@ GATEWAY_WS_URL=ws://gateway-node:8546
 TX_SENDER_PRIVATE_KEY=0x8f82b3f482c19a95ac29c82cf048c076ed0de2530c64a73f2d2d7d1e64b5cc6e
 INPUT_VERIFICATION_ADDRESS=0x3b12Fc766Eb598b285998877e8E90F3e43a1F8d2
 CIPHERTEXT_COMMITS_ADDRESS=0xeAC2EfFA07844aB326D92d1De29E136a6793DFFA
+GATEWAY_CONFIG_ADDRESS=0x576Ea67208b146E63C5255d0f90104E25e3e04c7
 KMS_GENERATION_ADDRESS=0x35760912360E875DA50D40a74305575c23D55783

test-suite/fhevm/fhevm-cli

@@ -80,7 +80,7 @@ function usage {
   echo -e "  ${YELLOW}deploy${RESET} ${CYAN}[--build] [--local] [--coprocessors N] [--coprocessor-threshold T]${RESET}    Deploy fhevm stack"
   echo -e "  ${YELLOW}pause${RESET} ${CYAN}[CONTRACTS]${RESET}     Pause specific contracts (host|gateway)"
   echo -e "  ${YELLOW}unpause${RESET} ${CYAN}[CONTRACTS]${RESET}     Unpause specific contracts (host|gateway)"
-  echo -e "  ${YELLOW}test${RESET} ${CYAN}[TYPE]${RESET}         Run tests (input-proof|user-decryption|public-decryption|delegated-user-decryption|random|random-subset|operators|erc20|hcu-block-cap|debug)"
+  echo -e "  ${YELLOW}test${RESET} ${CYAN}[TYPE]${RESET}         Run tests (input-proof|ciphertext-drift|user-decryption|public-decryption|delegated-user-decryption|random|random-subset|operators|erc20|hcu-block-cap|debug)"
   echo -e "  ${YELLOW}smoke${RESET} ${CYAN}[PROFILE]${RESET}      Run multicoproc smoke profile (multi-2-2|multi-3-5)"
   echo -e "  ${YELLOW}upgrade${RESET} ${CYAN}[SERVICE]${RESET}   Upgrade specific service (host|gateway|connector|coprocessor|relayer|test-suite)"
   echo -e "  ${YELLOW}clean${RESET}               Remove all containers and volumes"
@@ -102,6 +102,7 @@ function usage {
   echo -e "  ${PURPLE}./fhevm-cli smoke multi-2-2${RESET}"
   echo -e "  ${PURPLE}./fhevm-cli smoke multi-3-5${RESET}"
   echo -e "  ${PURPLE}./fhevm-cli test input-proof${RESET}"
+  echo -e "  ${PURPLE}./fhevm-cli test ciphertext-drift${RESET}"
   echo -e "  ${PURPLE}./fhevm-cli test input-proof --no-hardhat-compile${RESET}"
   echo -e "  ${PURPLE}./fhevm-cli test user-decryption ${RESET}"
   echo -e "  ${PURPLE}./fhevm-cli test public-decrypt-http-ebool ${RESET}"
@@ -350,6 +351,10 @@ case $COMMAND in
           log_message="${LIGHT_BLUE}${BOLD}[TEST] INPUT PROOF (uint64)${RESET}"
           docker_args+=("-g" "test add 42 to uint64 input and decrypt")
           ;;
+        ciphertext-drift)
+          echo -e "${LIGHT_BLUE}${BOLD}[TEST] CIPHERTEXT DRIFT${RESET}"
+          exec "${SCRIPT_DIR}/scripts/run-ciphertext-drift-e2e.sh"
+          ;;
         user-decryption)
           log_message="${LIGHT_BLUE}${BOLD}[TEST] USER DECRYPTION${RESET}"
           docker_args+=("-g" "test user decrypt")

test-suite/fhevm/scripts/inject-coprocessor-drift.sh

@@ -0,0 +1,46 @@
+#!/bin/bash
+set -euo pipefail
+
+INSTANCE_INDEX="${1:-1}"
+TIMEOUT_SECONDS="${DRIFT_INJECT_TIMEOUT_SECONDS:-180}"
+POLL_INTERVAL_SECONDS="${DRIFT_INJECT_POLL_INTERVAL_SECONDS:-2}"
+POSTGRES_CONTAINER="${POSTGRES_CONTAINER:-coprocessor-and-kms-db}"
+POSTGRES_USER="${POSTGRES_USER:-postgres}"
+POSTGRES_PASSWORD="${POSTGRES_PASSWORD:-postgres}"
+
+if ! [[ "$INSTANCE_INDEX" =~ ^[0-9]+$ ]]; then
+  echo "instance index must be a non-negative integer" >&2
+  exit 1
+fi
+
+db_name="coprocessor"
+if [ "$INSTANCE_INDEX" -gt 0 ]; then
+  db_name="coprocessor_${INSTANCE_INDEX}"
+fi
+
+psql_query() {
+  docker exec -e PGPASSWORD="$POSTGRES_PASSWORD" "$POSTGRES_CONTAINER" \
+    psql -U "$POSTGRES_USER" -d "$db_name" -t -A -c "$1"
+}
+
+baseline_handles="$(psql_query "SELECT encode(handle, 'hex') FROM ciphertext_digest;" || true)"
+deadline=$((SECONDS + TIMEOUT_SECONDS))
+
+while [ "$SECONDS" -lt "$deadline" ]; do
+  ready_handles="$(psql_query "SELECT encode(handle, 'hex') FROM ciphertext_digest WHERE txn_is_sent = false AND ciphertext IS NOT NULL AND ciphertext128 IS NOT NULL ORDER BY created_at DESC;")"
+  while IFS= read -r handle_hex; do
+    [ -z "$handle_hex" ] && continue
+    if printf '%s\n' "$baseline_handles" | grep -Fxq "$handle_hex"; then
+      continue
+    fi
+
+    psql_query "UPDATE ciphertext_digest SET ciphertext = set_byte(ciphertext, 0, get_byte(ciphertext, 0) # 1) WHERE handle = decode('${handle_hex}', 'hex') AND txn_is_sent = false;"
+    echo "$handle_hex"
+    exit 0
+  done <<< "$ready_handles"
+
+  sleep "$POLL_INTERVAL_SECONDS"
+done
+
+echo "timed out waiting for a new ready ciphertext_digest row in ${db_name}" >&2
+exit 1

test-suite/fhevm/scripts/run-ciphertext-drift-e2e.sh

@@ -0,0 +1,76 @@
+#!/bin/bash
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+FAULTY_INSTANCE_INDEX="${FAULTY_INSTANCE_INDEX:-1}"
+FAULTY_TX_SENDER_CONTAINER="${FAULTY_TX_SENDER_CONTAINER:-coprocessor${FAULTY_INSTANCE_INDEX}-transaction-sender}"
+TEST_CONTAINER="${TEST_CONTAINER:-fhevm-test-suite-e2e-debug}"
+GREP_PATTERN="${GREP_PATTERN:-test user input uint64 \\(non-trivial\\)}"
+METRIC_NAME="coprocessor_gw_listener_drift_detected_counter"
+METRIC_TIMEOUT_SECONDS="${DRIFT_METRIC_TIMEOUT_SECONDS:-180}"
+METRIC_POLL_INTERVAL_SECONDS="${DRIFT_METRIC_POLL_INTERVAL_SECONDS:-2}"
+HANDLE_FILE="$(mktemp)"
+injector_pid=""
+
+metric_total() {
+  local total=0
+  local container
+  while IFS= read -r container; do
+    [ -z "$container" ] && continue
+    local value
+    value="$(docker exec "$container" curl -fsS http://127.0.0.1:9100/metrics 2>/dev/null | awk -v metric="$METRIC_NAME" '$1 == metric {sum += $2} END {print sum + 0}')"
+    total=$((total + value))
+  done < <(docker ps --format '{{.Names}}' | grep -E '^coprocessor([0-9]+)?-gw-listener$' || true)
+  echo "$total"
+}
+
+wait_for_metric_increment() {
+  local baseline="$1"
+  local deadline=$((SECONDS + METRIC_TIMEOUT_SECONDS))
+  while [ "$SECONDS" -lt "$deadline" ]; do
+    local current
+    current="$(metric_total)"
+    if [ "$current" -gt "$baseline" ]; then
+      echo "$current"
+      return 0
+    fi
+    sleep "$METRIC_POLL_INTERVAL_SECONDS"
+  done
+  return 1
+}
+
+cleanup() {
+  if [ -n "$injector_pid" ] && kill -0 "$injector_pid" >/dev/null 2>&1; then
+    kill "$injector_pid" >/dev/null 2>&1 || true
+  fi
+  docker start "$FAULTY_TX_SENDER_CONTAINER" >/dev/null 2>&1 || true
+  rm -f "$HANDLE_FILE"
+}
+
+trap cleanup EXIT
+
+baseline_metric="$(metric_total)"
+docker stop "$FAULTY_TX_SENDER_CONTAINER" >/dev/null
+
+"${SCRIPT_DIR}/inject-coprocessor-drift.sh" "$FAULTY_INSTANCE_INDEX" > "$HANDLE_FILE" &
+injector_pid=$!
+
+docker exec \
+  -e GATEWAY_RPC_URL= \
+  -e CIPHERTEXT_COMMITS_ADDRESS= \
+  -e INPUT_VERIFICATION_ADDRESS= \
+  "$TEST_CONTAINER" \
+  ./run-tests.sh -n staging -g "$GREP_PATTERN"
+
+wait "$injector_pid"
+injector_pid=""
+handle_hex="$(cat "$HANDLE_FILE")"
+
+docker start "$FAULTY_TX_SENDER_CONTAINER" >/dev/null
+
+if ! updated_metric="$(wait_for_metric_increment "$baseline_metric")"; then
+  echo "drift metric did not increase after injecting handle ${handle_hex}" >&2
+  exit 1
+fi
+
+echo "drift detected for handle ${handle_hex} (${METRIC_NAME}: ${baseline_metric} -> ${updated_metric})"