Skip to content

Commit 974fe41

Browse files
authored
chore(coprocessor): remove legacy tfhe-worker gRPC path (#1982)
* chore(coprocessor): remove legacy tfhe-worker grpc path * fix(tfhe-worker): resolve clippy dead_code in bench/test utils * refactor(tfhe-worker): remove unused computation module * test(tfhe-worker): cap event operator coverage at uint64 * fix(coprocessor): address review noise and typos * chore(tfhe-worker): reduce bench fmt churn in dex migration * chore(tfhe-worker): revert formatting-only bench_id wraps * chore(tfhe-worker): remove remaining bench format-only churn * bench(tfhe-worker): restore dex workload parity with legacy grpc * test(tfhe-worker): restore non-ignored coverage after grpc removal * test(tfhe-worker): deduplicate operator event coverage * test(tfhe-worker): harden event test stability * test(tfhe-worker): run full event type matrix in CI * test(tfhe-worker): default full event matrix with mode logging * test(tfhe-worker): simplify event matrix selection * docs(tfhe-worker): document event test matrix modes * test(tfhe-worker): expand random event tests across types * test(tfhe-worker): restore random type matrix parity * test(tfhe-worker): use query! in invalid operation event test * fix(bench): stabilize benchmark pipeline after grpc refactor * fix(bench): allow dex setup trivial encrypt handles * charts: bump coprocessor chart version * tfhe-worker: propagate gpu feature to test-harness * test(tfhe-worker): allow dependent schedule setup handle * test(tfhe-worker): fix event test matrix CI regressions * refactor(tfhe-worker): deduplicate test helpers and remove dead code - Migrate operators_from_events.rs to use shared event_helpers (setup_event_harness, next_handle, to_ty, tfhe_event, log_with_tx) - Remove duplicate test_invalid_operation_marks_error (kept in errors.rs) - Move wait_for_error to event_helpers for shared use - Extract TEST_CHAIN_ID const, remove debug eprintln calls - Remove 16 dead CoprocessorError variants from types.rs * refactor(tfhe-worker): destructure EventHarness to reduce PR diff Destructure setup_event_harness() return into {app, pool, listener_db} so variable names match the original code, minimising the review diff. * chore(tfhe-worker): remove dead deps and batch event test waits Remove 6 Cargo dependencies that were only used by the deleted gRPC server (sha3, lru, rayon, tfhe-zk-pok, regex, actix-web). Restructure 4 event tests (unary, cast, if-then-else, rand) to use batch-then-wait pattern: insert all events first, call wait_until_all_allowed_handles_computed once, then verify. This eliminates ~200 redundant waits in CI, saving ~10 minutes of sleep. Also remove unnecessary pub(super) from test_fhe_rand_events. * refactor(tfhe-worker): address PR review feedback - Upgrade as_scalar_uint to accept &BigInt directly - Deduplicate helpers in operators_from_events.rs (delete insert_tfhe_event, allow_handle, as_scalar_uint copies; use event_helpers versions) - Delete redundant test_fhe_rand_events (subset of random.rs tests) - Expand test_op_trivial_encrypt to cover all supported types with edge-case values - Add 5 error test scenarios: circular dependency, too many inputs, scalar division by zero, binary boolean inputs, unary boolean inputs * fix(tfhe-worker): replace validation-time error tests with execution-time ones Remove 3 error tests (circular dependency, too many inputs, scalar div by zero) that trigger validation-time errors in check_fhe_operand_types. These errors propagate via ? without being persisted to the DB, causing an infinite retry loop in event-driven mode. Replace with test_type_mismatch_error (FheAdd on uint8 + uint16) which passes validation but properly fails at execution time with UnsupportedFheTypes. The validation-path error propagation is tracked as a separate issue. * docs: update FHE computation diagram to reflect event-driven architecture Replace the obsolete AsyncCompute gRPC flow with the current host-listener event-driven architecture in the sequence diagram. * fix(tfhe-worker): fix GPU test failures in error and random bounded tests test_coprocessor_computation_errors: Replace Cast-to-type-255 with FheSub on mismatched types (uint32 + uint64). The old test panicked on the GPU path during memory reservation in trivial_encrypt_be_bytes, preventing the error from being persisted to the DB. Type-mismatch errors return a proper Result::Err on both CPU and GPU. test_fhe_random_bounded: Use per-type bounds from the old gRPC test instead of upper_bound=1. The 0-random-bits edge case (bound=1) behaves differently on GPU vs CPU. Also check bool results as true/false rather than assuming a specific numeric value, since CPU and GPU produce different deterministic outputs for the same seed. * docs(tfhe-worker): fix stale README heading after gRPC removal The server was removed; only the background worker remains.
1 parent 2458fa9 commit 974fe41

33 files changed

+3422
-9479
lines changed

charts/coprocessor/Chart.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
name: coprocessor
22
description: A helm chart to distribute and deploy Zama fhevm Co-Processor services
3-
version: 0.8.2
3+
version: 0.8.3
44
apiVersion: v2
55
keywords:
66
- fhevm

charts/coprocessor/values.yaml

Lines changed: 1 addition & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -535,15 +535,8 @@ tfheWorker:
535535
- --metrics-addr=0.0.0.0:9100
536536
- --service-name=tfhe-worker # tfhe-worker service name in OTLP traces
537537
- --log-level=INFO
538-
# Should not be used (unsafe - testing only, keep 0/false values except CI)
539-
- --run-server=false
538+
# Should not be used (unsafe - testing only, keep false values except CI)
540539
- --generate-fhe-keys=false
541-
- --server-maximum-ciphertexts-to-schedule=0
542-
- --server-maximum-ciphertexts-to-get=0
543-
- --maximum-compact-inputs-upload=0
544-
- --maximum-handles-per-input=0
545-
- --server-addr=""
546-
- --coprocessor-private-key=""
547540
# Unique worker identifier (valid UUID v4 format)
548541
# If not set, defaults to a random UUID generated at startup
549542
- --worker-id=$(WORKER_ID)

coprocessor/README.md

Lines changed: 0 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -72,38 +72,24 @@ $ tfhe_worker --help
7272
Usage: tfhe_worker [OPTIONS]
7373

7474
Options:
75-
--run-server
76-
Run the API server
7775
--run-bg-worker
7876
Run the background worker
7977
--generate-fhe-keys
8078
Generate fhe keys and exit
81-
--server-maximum-ciphertexts-to-schedule <SERVER_MAXIMUM_CIPHERTEXTS_TO_SCHEDULE>
82-
Server maximum ciphertexts to schedule per batch [default: 5000]
83-
--server-maximum-ciphertexts-to-get <SERVER_MAXIMUM_CIPHERTEXTS_TO_GET>
84-
Server maximum ciphertexts to serve on get_cihpertexts endpoint [default: 5000]
8579
--work-items-batch-size <WORK_ITEMS_BATCH_SIZE>
8680
Work items batch size [default: 10]
8781
--tenant-key-cache-size <TENANT_KEY_CACHE_SIZE>
8882
Tenant key cache size [default: 32]
89-
--maximum-compact-inputs-upload <MAXIMUM_COMPACT_INPUTS_UPLOAD>
90-
Maximum compact inputs to upload [default: 10]
91-
--maximum-handles-per-input <MAXIMUM_HANDLES_PER_INPUT>
92-
Maximum compact inputs to upload [default: 255]
9383
--coprocessor-fhe-threads <COPROCESSOR_FHE_THREADS>
9484
Coprocessor FHE processing threads [default: 8]
9585
--tokio-threads <TOKIO_THREADS>
9686
Tokio Async IO threads [default: 4]
9787
--pg-pool-max-connections <PG_POOL_MAX_CONNECTIONS>
9888
Postgres pool max connections [default: 10]
99-
--server-addr <SERVER_ADDR>
100-
Server socket address [default: 127.0.0.1:50051]
10189
--metrics-addr <METRICS_ADDR>
10290
Prometheus metrics server address [default: 0.0.0.0:9100]
10391
--database-url <DATABASE_URL>
10492
Postgres database url. If unspecified DATABASE_URL environment variable is used
105-
--coprocessor-private-key <COPROCESSOR_PRIVATE_KEY>
106-
Coprocessor private key file path. Private key is in plain text 0x1234.. format [default: ./coprocessor.key]
10793
```
10894

10995
```bash

coprocessor/docs/fundamentals/fhevm/coprocessor/fhe_computation.md

Lines changed: 12 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -5,30 +5,32 @@ Block execution in FHEVM-coprocessor is split into two parts:
55
- Symbolic Execution (onchain)
66
- FHE Computation (offchain)
77

8-
Symbolic execution happens onchain, inside the [FHEVMExecutor](../../../../contracts/contracts/FHEVMExecutor.sol) contract (inside the EVM). Essentially, the EVM accumulates all requested FHE operations in a block with their input handles and the corresponding result handles. Then, at the end of block execution, it sends an `AsyncCompute` request to the coprocessor such that FHE computation can be done **eventually**. Note that FHE computation can be done at a future point in time, after the block has been committed on the host blockchain. We can do that, symbolic execution only needs handles and doesn't actual FHE ciphertexts. Actual FHE ciphertexts are needed only on **decryption** and **reencryption**, i.e. when a user wants to see the plaintext value.
8+
Symbolic execution happens onchain, inside the [FHEVMExecutor](../../../../contracts/contracts/FHEVMExecutor.sol) contract (inside the EVM). Essentially, the EVM accumulates all requested FHE operations in a block with their input handles and the corresponding result handles. These operations are emitted as on-chain events (logs) that the host-listener ingests into the coprocessor database, such that FHE computation can be done **eventually**. Note that FHE computation can be done at a future point in time, after the block has been committed on the host blockchain. We can do that, symbolic execution only needs handles and doesn't need actual FHE ciphertexts. Actual FHE ciphertexts are needed only on **decryption** and **reencryption**, i.e. when a user wants to see the plaintext value.
99

1010
```mermaid
1111
sequenceDiagram
1212
participant Full Node
13-
participant Coprocessor
13+
participant Host Listener
1414
participant DB
15+
participant TFHE Worker
1516
1617
loop Block Execution - Symbolic
1718
Note over Full Node: Symbolic Execution on handles in Solidity
1819
Note over Full Node: Inside EVM: computations.add(op, [inputs], [result_handles])
1920
end
2021
2122
Note over Full Node: End of Block Execution
23+
Note over Full Node: FHE operations emitted as on-chain events (logs)
2224
23-
Full Node->>+Coprocessor: AsyncCompute (AsyncComputeRequest(computations))
24-
Coprocessor->>+DB: Insert Computations
25-
DB->>-Coprocessor: Ack
26-
Coprocessor->>-Full Node: AsyncComputeResponse
25+
Host Listener->>Full Node: Poll for new events
26+
Full Node->>Host Listener: FHE operation events
27+
Host Listener->>+DB: Insert Computations
28+
DB->>-Host Listener: Ack
2729
2830
loop FHE Computation
29-
Coprocessor --> DB: Read Input Ciphertexts
30-
Note over Coprocessor: FHE Computation
31-
Coprocessor --> DB: Write Result Ciphertexts
31+
TFHE Worker --> DB: Read Input Ciphertexts
32+
Note over TFHE Worker: FHE Computation
33+
TFHE Worker --> DB: Write Result Ciphertexts
3234
end
3335
```
3436

@@ -38,6 +40,6 @@ Note that, for now, we omit the Data Availability (DA) layer. It is still work i
3840

3941
## Parallel Execution
4042

41-
Since the Coprocessor can extract data dependencies from the `AsyncCompute` request, it can use them to execute FHE computations in parallel.
43+
Since the coprocessor can extract data dependencies from the ingested events, it can use them to execute FHE computations in parallel.
4244

4345
At the time of writing, the Coprocessor uses a simple policy to schedule FHE computation on multiple threads. More optimal policies will be introduced in the future and made configurable.

coprocessor/docs/getting_started/fhevm/coprocessor/configuration.md

Lines changed: 0 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -11,38 +11,24 @@ coprocessor --help
1111
Usage: coprocessor [OPTIONS]
1212
1313
Options:
14-
--run-server
15-
Run the API server
1614
--run-bg-worker
1715
Run the background worker
1816
--generate-fhe-keys
1917
Generate fhe keys and exit
20-
--server-maximum-ciphertexts-to-schedule <SERVER_MAXIMUM_CIPHERTEXTS_TO_SCHEDULE>
21-
Server maximum ciphertexts to schedule per batch [default: 5000]
22-
--server-maximum-ciphertexts-to-get <SERVER_MAXIMUM_CIPHERTEXTS_TO_GET>
23-
Server maximum ciphertexts to serve on get_cihpertexts endpoint [default: 5000]
2418
--work-items-batch-size <WORK_ITEMS_BATCH_SIZE>
2519
Work items batch size [default: 10]
2620
--tenant-key-cache-size <TENANT_KEY_CACHE_SIZE>
2721
Tenant key cache size [default: 32]
28-
--maximum-compact-inputs-upload <MAXIMUM_COMPACT_INPUTS_UPLOAD>
29-
Maximum compact inputs to upload [default: 10]
30-
--maximum-handles-per-input <MAXIMUM_HANDLES_PER_INPUT>
31-
Maximum compact inputs to upload [default: 255]
3222
--coprocessor-fhe-threads <COPROCESSOR_FHE_THREADS>
3323
Coprocessor FHE processing threads [default: 8]
3424
--tokio-threads <TOKIO_THREADS>
3525
Tokio Async IO threads [default: 4]
3626
--pg-pool-max-connections <PG_POOL_MAX_CONNECTIONS>
3727
Postgres pool max connections [default: 10]
38-
--server-addr <SERVER_ADDR>
39-
Server socket address [default: 127.0.0.1:50051]
4028
--metrics-addr <METRICS_ADDR>
4129
Prometheus metrics server address [default: 0.0.0.0:9100]
4230
--database-url <DATABASE_URL>
4331
Postgres database url. If unspecified DATABASE_URL environment variable is used
44-
--coprocessor-private-key <COPROCESSOR_PRIVATE_KEY>
45-
Coprocessor private key file path. Private key is in plain text 0x1234.. format [default: ./coprocessor.key]
4632
--service-name <SERVICE_NAME>
4733
Coprocessor service name in OTLP traces [default: coprocessor]
4834
-h, --help
@@ -61,19 +47,3 @@ Note that there are two thread pools in the Coprocessor backend:
6147
The tokio one (set via `--tokio-threads`) determines how many tokio threads are spawned. These threads are used for async tasks and should not be blocked.
6248

6349
The FHE compute threads are the ones that actually run the FHE computation (set via `--coprocessor-fhe-threads`).
64-
65-
#### Secret Signing Key
66-
67-
A secret signing key is needed to allow the Coprocessor backend sign input insertion requests. A `coprocessor.key` file can be given on the command line for the **server** as:
68-
69-
```
70-
coprocessor --help
71-
Usage: coprocessor [OPTIONS]
72-
73-
Options:
74-
...
75-
--coprocessor-private-key <COPROCESSOR_PRIVATE_KEY>
76-
Coprocessor private key file path. Private key is in plain text 0x1234.. format [default: ./coprocessor.key]
77-
```
78-
79-
The secret signing key must be kept safe when operating the Coprocessor.

coprocessor/fhevm-engine/.sqlx/query-9f94cefdbf0af3d9e402d4dd717aa8eaeec85011ca6eceae762dc6a28694ea80.json

Lines changed: 22 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

coprocessor/fhevm-engine/.sqlx/query-ad5fda5d82e0fd04a7f46de69922a3aea6493653b145a40b3c04bfc52ebc2a4b.json

Lines changed: 29 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)