feat(coprocessor): slow lane for dependent ops (#1907)

* feat(coprocessor): throttle dependent ops

* feat(coprocessor): slow lane for dependent ops

* refactor(coprocessor): decide slow lane per chain

* Count dependent ops on insert

* refactor(coprocessor): add schedule lane enum

* feat(coprocessor): per-caller dependent ops limiter

* feat(coprocessor): weight dependent ops by cost

* feat(coprocessor): add per-chain caps

* refactor(coprocessor): rename schedule lane to priority

* feat(coprocessor): make priority continuous

* fix(tfhe-worker): keep slow-lane priority monotonic

* feat(host-listener): align op weights with HCU

* refactor: make schedule priority binary

* refactor(host-listener): drop distinct-caller cap

* refactor(host-listener): remove per-caller limiter

* refactor(host-listener): simplify slow-lane inputs

* refactor(host-listener): remove unused tfhe_caller

* refactor(host-listener): widen dependent ops counters

* fix(host-listener): harden slow-lane disable path

* fix(host-listener): bound slow-priority reset load in off mode

* refactor(host-listener): simplify off-mode to promote seen chains

* chore(charts): expose dependent ops cap in coprocessor values

* fix(coprocessor): bump chart and simplify slow-lane op weights

* refactor(host-listener): use unweighted dependent-op caps

* chore(coprocessor): refresh sqlx cache after rebase

* test(coprocessor): add contention e2e and chart guard

* test(host-listener): simplify slow-lane scenario fixtures

* docs(host-listener): add local slow-lane validation runbook

* test(host-listener): harden slow-lane local validation

* refactor(host-listener): clarify slow-lane naming

* chore(host-listener): add auditable slow-lane validation runbook

* test-suite: harden slow-lane validator bootstrap checks

* test(host-listener): make slow-lane local validation deterministic

* fix(tfhe-worker): preserve fast-lane order in early lock

* refactor(host-listener): simplify slow-lane ingest classification

* test(host-listener): assert schedule-priority migration contract

* refactor(host-listener): use set-based slow-chain classification

* refactor(host-listener): drop dependent-op counters and simplify classification

* feat(host-listener): add lane-level queue and marking metrics

* refactor(host-listener): drop lane queue DB metrics from hot path

* fix(tfhe-worker): add starvation escape hatch and trim validation docs

* refactor(host-listener): use explicit expect for metric registration

* fix(host-listener): inherit slow lane from slow parent chains

* refactor(slow-lane): tighten priority typing and cleanup

* refactor(slow-lane): remove unused priority TryFrom

* refactor(host-listener): use eager slow-lane reset in off mode

* fix(host-listener): guard eager reset with advisory lock

* fix(host-listener): block on eager reset advisory lock

* docs(host-listener): add slow-lane testnet incident runbook

* refactor(tfhe-worker): simplify fallback by making early lock lane-agnostic

* docs(host-listener): switch slow-lane runbook to metrics/logs gates

* refactor(host-listener): use sqlx query! for slow-lane DB helpers

* fix(host-listener): use execute for advisory lock query

* test(host-listener): add slow-lane inheritance and monotonicity coverage

* perf(host-listener): batch off-mode slow-lane promotion

* feat(host-listener): use inheritance-only parent metadata for slow lane

* refactor(host-listener): dedupe slow-parent ids in DB helper

* Revert "refactor(host-listener): dedupe slow-parent ids in DB helper"

This reverts commit a9c181e.

* docs(host-listener): clarify off-mode batched startup promotion

* docs(host-listener): clarify inheritance_parents is ingest-only

* test(host-listener): remove stale chain-id test and stabilize reorg assertions

* fix(host-listener): resolve rebase regressions in imports and test DB init

* fix(charts): bump coprocessor chart version to 0.8.1

* chore(host-listener): drop out-of-scope schedule_order extraction

Also fix clippy checked_conversions in chain_id for pre-commit.

* docs(host-listener): clarify dependent-ops cap semantics and tuning

* fix(host-listener): count all dependent ops for slow-lane cap

Drop is_allowed filter to avoid undercounting required producer ops.

* docs(host-listener): clarify non-allowed ops can still be required

* docs(host-listener): make slow-lane flag section concise

* docs(host-listener): trim slow-lane runbook and local notes

* fix(host-listener): count all inserted ops for slow-lane cap

Use inserted-only per-chain counting to avoid underestimating required producer pressure.

* test(host-listener): align slow-lane threshold matrix with inserted-only counting

* feat(host-listener): account slow-lane over split dependency closures

Keep no_fork parallelism, but aggregate inserted-op pressure over split dependency closures for slow-lane classification.

* docs(host-listener): clarify split-closure slow-lane classifier

* docs(host-listener): simplify split-dependency classifier comment

* refactor(host-listener): namespace slow-lane advisory lock by chain

* refactor(host-listener): remove cached chain_id label from Database

* refactor(host-listener): simplify inserted-op counter update

Author: Eikix

charts/coprocessor/Chart.yaml

@@ -1,6 +1,6 @@
 name: coprocessor
 description: A helm chart to distribute and deploy Zama fhevm Co-Processor services
-version: 0.8.0
+version: 0.8.1
 apiVersion: v2
 keywords:
   - fhevm

charts/coprocessor/templates/coprocessor-host-listener-catchup-only-deployment.yaml

@@ -47,7 +47,12 @@ spec:
           image: {{ .Values.hostListenerCatchupOnly.image.name }}:{{ .Values.hostListenerCatchupOnly.image.tag }}
           command: ["host_listener"]
           args:
-{{ toYaml .Values.hostListenerCatchupOnly.args | nindent 12 }}
+            {{- range $arg := .Values.hostListenerCatchupOnly.args }}
+            {{- if not (hasPrefix "--dependent-ops-max-per-chain=" $arg) }}
+            - {{ $arg | quote }}
+            {{- end }}
+            {{- end }}
+            - --dependent-ops-max-per-chain={{ int .Values.dependentOps.maxPerChain }}
           env:
 {{ toYaml .Values.hostListenerCatchupOnly.env | nindent 12 }}
           ports:

charts/coprocessor/templates/coprocessor-host-listener-deployment.yaml

@@ -47,7 +47,12 @@ spec:
           image: {{ .Values.hostListener.image.name }}:{{ .Values.hostListener.image.tag }}
           command: ["host_listener"]
           args:
-{{ toYaml .Values.hostListener.args | nindent 12 }}
+            {{- range $arg := .Values.hostListener.args }}
+            {{- if not (hasPrefix "--dependent-ops-max-per-chain=" $arg) }}
+            - {{ $arg | quote }}
+            {{- end }}
+            {{- end }}
+            - --dependent-ops-max-per-chain={{ int .Values.dependentOps.maxPerChain }}
           env:
 {{ toYaml .Values.hostListener.env | nindent 12 }}
           ports:

charts/coprocessor/templates/coprocessor-host-listener-poller-deployment.yaml

@@ -47,7 +47,12 @@ spec:
           image: {{ .Values.hostListenerPoller.image.name }}:{{ .Values.hostListenerPoller.image.tag }}
           command: ["host_listener_poller"]
           args:
-{{ toYaml .Values.hostListenerPoller.args | nindent 12 }}
+            {{- range $arg := .Values.hostListenerPoller.args }}
+            {{- if not (hasPrefix "--dependent-ops-max-per-chain=" $arg) }}
+            - {{ $arg | quote }}
+            {{- end }}
+            {{- end }}
+            - --dependent-ops-max-per-chain={{ int .Values.dependentOps.maxPerChain }}
           env:
 {{ toYaml .Values.hostListenerPoller.env | nindent 12 }}
           ports:

charts/coprocessor/values.yaml

@@ -1,6 +1,14 @@
 # =============================================================================
 # FHEVM Coprocessor Configuration
 # =============================================================================
+
+# -----------------------------------------------------------------------------
+# Shared host-listener dependence settings (applies to all HL types)
+# -----------------------------------------------------------------------------
+dependentOps:
+  # Max dependent ops per chain per ingested block before slow-lane.
+  # 0 disables slow-lane decisions.
+  maxPerChain: 0
 # This chart deploys the FHEVM coprocessor components including:
 # - Configuration setup
 # - Database migration
@@ -135,6 +143,7 @@ hostListener:
     # - --dependence-cache-size=10000
     # - --dependence-by-connexity   # Whether to build connected components or linear chains (default no)
     # - --dependence-cross-block    # Do chains cross L1 block boundaries (default yes)
+    # --dependent-ops-max-per-chain is injected from dependentOps.maxPerChain
 
     ### Catchup parameters (optional)
     # - --catchup-margin
@@ -247,6 +256,7 @@ hostListenerPoller:
     # - --dependence-cache-size=10000
     # - --dependence-by-connexity   # Whether to build connected components or linear chains (default no)
     # - --dependence-cross-block    # Do chains cross L1 block boundaries (default yes)
+    # --dependent-ops-max-per-chain is injected from dependentOps.maxPerChain
 
   # Service ports configuration
   ports:
@@ -342,6 +352,7 @@ hostListenerCatchupOnly:
     # - --start-at-block  # To catch up from a specific block
     # - --catchup-margin
     # - --catchup-paging
+    # --dependent-ops-max-per-chain is injected from dependentOps.maxPerChain
 
   # Service ports configuration
   ports: