feat(coprocessor): process work along dependence chains (#1718)

* feat(coprocessor): schedule computations along dependence chains in coprocessor (#1550)

* feat(coprocessor): create dependence_chain table

* feat(coprocessor): coordinate dependence-chain processing across multiple workers

It provides a non-blocking, distributed locking mechanism that
coordinates dependence-chain processing across multiple tfhe-workers.

A worker can acquire ownership of the next available dependence-chain entry for processing
ordered by last_updated_at (FIFO queue-like approach).

Ownership expires after a timeout, enabling work-stealing by other workers.

New CLI param --worker_id

* test(coprocessor): ensure both acquire_next_lock and work-stealing features

* fix(coprocessor): fix work-stealing when a lock has expired

- Added LockingReason for logging
- Make expiry configurable

* fix(coprocessor): update the flow of acquire/extend/release chain_id lock

* chore(coprocessor): update sqlx cache

* chore(coprocessor): improve logging for dcid locking

* chore(coprocessor): disable fallback for dependence_chain_id locking

* fix(coprocessor): update in-memory lock info on extend_current_lock

* fix(coprocessor): lock another dcid and continue processing

* chore(coprocessor): update sqlx cache

* chore(coprocessor): add idx_dependence_chain_processing_by_worker

* chore(coprocessor): observe query timings in tfhe-worker

- add --dcid_ttl_sec config
- add otel traces for dcid

* chore(coprocessor): implement both max_lock_ttl_sec and disable_dcid_locking options

* chore(coprocessor): update sqlx cache

* chore(coprocessor): update last_updated_at when releasing a lock

* chore(coprocessor): support --dcid-timeslice-sec CLI param, tfhe-worker

* chore(coprocessor): solve dcid unit-tests issue

* chore(coprocessor): enable lock re-acquisition once the timeslice has been exceeded

* chore(coprocessor): enable default timeslice

* chore(coprocessor): run cleanup procedure to delete old processed dcids

* chore(coprocessor): acquire locks only on DCIDs that are ready for computation

* chore(coprocessor): update charts with new tfhe-worker args

* chore(coprocessor): handle case no-dcid-available

* chore(coprocessor): bump chart version

* chore(coprocessor): notify work available if dependency count reaches zero

* fix(coprocessor): add dependence chain index on last_updated when processed

* fix(coprocessor): update is_completed only where a CT is inserted in DB

* fix(coprocessor): restrict update of computation completion to uncompleted computations

* fix(coprocessor): prevent dependence cycle overestimation on trivial encrypt handles

* fix(coprocessor): update test for completion of processing of dcid

* fix(coprocessor): add missing partial indexes on ciphertexts and ciphertext_digest tables

* feat(coprocessor): add transaction dependence chains in HL (#1651)

* fix(coprocessor): host-listener, dependency chain

* fix(coprocessor): fix to squash, duplicated trivial encrypt

* fix(coprocessor): fix to squash, duplicated trivial encrypt, test

* fix(coprocessor): fix to squash, scalars are not handles

* fix(coprocessor): fix to squash, cargo fmt

* feat(coprocessor): topologic timestamp

* fix(coprocessor): host-listener, reject cycle and describe out of order dependencies

* fix(coprocessor): do not update dependence chain timestamp on row update

* fix(coprocessor): host-listener, bad condition for need to sort tx

* feat(coprocessor): host-listener, dependency_count for dependency_chain

* feat(coprocessor): host-listener, dependents for dependency_chain

* fix(coprocessor): restrict dependence counter to block scope

* fix(coprocessor): do not update dependence chain last_updated_at on release

* fix(coprocessor): emit warning only when dependence chain is missing dependences

* feat(coprocessor): host-listener, dependency_chain as connected component

* fix(coprocessor): host-listener, update last_updated_at de chain when already processed

* fix(coprocessor): deprecate schedule order in TFHE worker

* chore(coprocessor): fix CI

* fix(coprocessor): host-listener, params for dependency chain policy

* fix(coprocessor): hist-listener, dependency_chain, cycle detection

* chore(coprocessor): update charts for new params

* chore(coprocessor): fix TFHE worker CI test test_extend_or_release_lock_2

---------

Co-authored-by: Antoniu Pop <antoniu.pop@zama.ai>
Co-authored-by: Antoniu Pop <90181190+antoniupop@users.noreply.github.com>
Co-authored-by: rudy <rudy.sicard@zama.ai>

* fix(coprocessor): db migration, improve indexing for sns worker fetching work (#1692)

* fix(coprocessor): db migration, improve indexing for sns worker fetching work

* fix(coprocessor): add missing indexes for selecting allowed handles when tx unsent

---------

Co-authored-by: Antoniu Pop <antoniu.pop@zama.ai>

* feat(coprocessor): add mechanism to release dependence chains when no progress (#1696)

* fix(coprocessor): do not update is_completed on unallowed handles

* feat(coprocessor): add mechanism to release dependence chains when no progress

* fix(coprocessor): remove obsolete row lock on computations

* feat(coprocessor): set created_at as topological order within block

* fix(coprocessor): chain release and update

* chore(coprocessor): update charts

* fix(coprocessor): fix top timestamp for tx

* fix(coprocessor): update earliest schedule order

* fix(coprocessor): remove adding epsilon to timestamp when releasing chain

* fix(coprocessor): split dependence chains after forks instead of before

---------

Co-authored-by: rudy <rudy.sicard@zama.ai>

* fix(coprocessor): add missing indexes on verify_proofs and dependence_chain tables (#1715)

* fix(coprocessor): db-migration, first clean on more obvious unused index (#1722)

* fix(coprocessor): align host listener and poller dependence params (#1728)

---------

Co-authored-by: goshawk-3 <76947196+goshawk-3@users.noreply.github.com>
Co-authored-by: rudy <rudy.sicard@zama.ai>

Author: 3 people

.github/workflows/test-suite-e2e-operators-tests.yml

@@ -101,6 +101,13 @@ jobs:
           username: ${{ github.actor }}
           password: ${{ secrets.GHCR_READ_TOKEN }}
 
+      - name: Login to Chainguard Registry
+        uses: docker/login-action@9780b0c442fbb1117ed29e0efdff1e18412f7567 # v3.3.0
+        with:
+          registry: cgr.dev
+          username: ${{ secrets.CGR_USERNAME }}
+          password: ${{ secrets.CGR_PASSWORD }}
+
       - name: Deploy fhevm Stack
         working-directory: test-suite/fhevm
         env:

charts/coprocessor/Chart.yaml

@@ -1,6 +1,6 @@
 name: coprocessor
 description: A helm chart to distribute and deploy Zama fhevm Co-Processor services
-version: 0.7.8
+version: 0.7.10
 apiVersion: v2
 keywords:
   - fhevm

charts/coprocessor/values.yaml

@@ -153,9 +153,13 @@ hostListener:
     - --initial-block-time=12  # it can switch to real blockTime when data is available
     - --log-level=INFO
     - --health-port=8080
-    - --dependence-cache-size=128
     - --reorg-maximum-duration-in-blocks=50
 
+    ### Dependence chains parameters
+    # - --dependence-cache-size=10000
+    # - --dependence-by-connexity   # Whether to build connected components or linear chains (default no)
+    # - --dependence-cross-block    # Do chains cross L1 block boundaries (default yes)
+
     ### Catchup parameters (optional)
     # - --catchup-margin
     # - --catchup-paging
@@ -266,6 +270,11 @@ hostListenerPoller:
     ### Prometheus metrics
     - --metrics-addr=0.0.0.0:9100  # Address for Prometheus metrics HTTP server
 
+    ### Dependence chains parameters
+    # - --dependence-cache-size=10000
+    # - --dependence-by-connexity   # Whether to build connected components or linear chains (default no)
+    # - --dependence-cross-block    # Do chains cross L1 block boundaries (default yes)
+
   # Service ports configuration
   ports:
     metrics: 9100
@@ -545,7 +554,7 @@ tfheWorker:
     - --run-bg-worker=true
     - --worker-polling-interval-ms=10000
     - --work-items-batch-size=100  # scheduling changed
-    - --dependence-chains-per-batch=100
+    - --dependence-chains-per-batch=100  # Deprecated. To be removed in a future release.
     - --tenant-key-cache-size=32
     - --coprocessor-fhe-threads=64  # scheduling changed
     - --tokio-threads=16  # scheduling changed
@@ -562,6 +571,23 @@ tfheWorker:
     - --maximum-handles-per-input=0
     - --server-addr=""
     - --coprocessor-private-key=""
+    # Unique worker identifier (valid UUID v4 format)
+    # If not set, defaults to a random UUID generated at startup
+    - --worker-id=$(WORKER_ID)
+    - --dcid-ttl-sec=30  # Time-to-live (in seconds) for dependence chain locks
+    # Disable dependence chain ID locking
+    # WARNING: May cause multiple workers to process the same DCID concurrently
+    # Defaults to false
+    - --disable-dcid-locking=false
+    # Time slice (in seconds) for processing a single dependence chain
+    # Locks are released if processing exceeds this duration
+    - --dcid-timeslice-sec=90
+    # Processed DCIDs older than this value are cleaned up
+    # Defaults to 48 hours (172800 seconds)
+    # Time-to-live (in seconds) for processed dependence chains
+    - --processed-dcid-ttl-sec=172800
+    - --dcid-cleanup-interval-sec=3600  # Interval (in seconds) for cleaning up expired DCID locks
+    - --dcid-max-no-progress-cycles=2   # Worker cycles without progress before releasing
 
   # Service ports configuration
   ports: