feat(coprocessor): standardize tracing/OTEL spans across workers (#1976)

* feat(coprocessor): migrate Phases C/D/H/J spans to tracing

Migrate remaining coprocessor workers from legacy OtelTracer to the
tracing + OTEL bridge pattern established in Phase A/B:

- Phase C: zkproof-worker verify_task spans (verifier.rs)
- Phase D: tfhe-worker high-signal execution loop spans (tfhe_worker.rs)
- Phase H: host-listener DB propagation spans (tfhe_event_propagate.rs)
- Phase J: transaction-sender op spans (add_ciphertext, allow_handle,
  delegate_user_decrypt, verify_proof)

Also cleans up telemetry.rs by removing the now-unused OtelTracer
wrapper and legacy span helpers, fixes a pre-existing clippy lint in
chain_id.rs, and fixes minor sns-worker span hygiene from Phase B.

Closes zama-ai/fhevm-internal#935
Closes zama-ai/fhevm-internal#1008
Closes zama-ai/fhevm-internal#1009
Closes zama-ai/fhevm-internal#1034
Closes zama-ai/fhevm-internal#1036

* refactor(coprocessor): use idiomatic #[tracing::instrument] everywhere

Address PR review feedback:

- Replace manual info_span!() + .enter() with #[tracing::instrument]
  on functions (host-listener, transaction-sender, zkproof-worker,
  tfhe-worker)
- Remove all set_parent() calls in zkproof-worker — tracing-opentelemetry
  layer handles parent propagation automatically via #[instrument]
- Remove span: tracing::Span parameter threading in verify_proof and
  create_ciphertext
- Use .instrument(loop_span) for async parent context in tfhe-worker
  instead of passing loop_span as a function parameter
- Revert out-of-scope chain_id.rs clippy fix

* refactor(coprocessor): remove last non-cross-boundary set_parent, document remaining

- Replace set_parent(t.context()) with parent: &t in batch_store span
  (same execution context, not cross-boundary)
- Add comments on the two remaining set_parent calls explaining they
  restore OTel context across async/thread boundaries

* refactor(coprocessor): migrate scheduler crate from raw OTel to tracing

Replace opentelemetry::global::tracer + manual spans with
tracing::info_span! in the scheduler's DFG execution pipeline.
Remove set_txn_id / BoxedSpan from telemetry.rs (no remaining callers).

* refactor(coprocessor): address review feedback on tracing style

* fix(coprocessor): restore tracing telemetry parity

* refactor(tfhe-worker): avoid unnecessary txn id allocation

* fix(coprocessor): address telemetry rereview cleanup items

* fix(coprocessor): address obatirou telemetry review comments

* refactor(coprocessor): dedupe short id telemetry helper

* refactor(coprocessor): streamline span metadata per obatirou review

* fix(zkproof-worker): instrument db_insert span timing

* fix(transaction-sender): restore prepare_delegate tracing parity

* fix(telemetry): simplify short-id helper callsites

* fix(coprocessor): drop out-of-scope chain_id style delta

* refactor(coprocessor): dedupe span error helper and remove stale dead_code attr

* fix(coprocessor): restore service otel wiring and align span op names

* refactor(coprocessor): simplify OTLP subscriber init fallback

* chore(coprocessor): remove unused init_otel helper

* fix(zkproof-worker): drop duplicate create_handle info event

* docs(telemetry): clarify OTLP fallback semantics

* refactor(telemetry): rename otel init helper for explicit fallback semantics

* refactor(coprocessor): unify info macro usage and remove duplicate event

* docs(coprocessor): add telemetry style guide and keep local OTLP endpoint

* refactor(tfhe-worker): use shared span-error helper in error paths

* refactor(txn-sender): simplify optional txn_id span fielding

* refactor(txn-sender): restore lean optional txn_id span recording

* fix(coprocessor): restore span attribute parity and fix A/B regressions

- Remove redundant info! events in tfhe-worker that duplicated span fields
- Restore db_insert count/valid and verify_proof list_len as span fields
  instead of log events in zkproof-worker (attribute parity with main)
- Add count field to expand_verified_list span (was set by caller on main)
- Restore distinct span names in txn-sender via #[instrument(name = ...)]
  to avoid collapsing 4 operations into 2 generic names

* refactor(coprocessor): remove redundant operation= span attributes

The span name already serves as the operation identifier in Jaeger/OTEL,
making the duplicate operation field unnecessary. Where operation differed
from the function name, name= on #[instrument] was already set in the
prior commit. The scheduler's FheGetCiphertext operation is intentionally
kept as it represents the FHE opcode, not the function name.

Author: Eikix

coprocessor/README.md

@@ -283,6 +283,55 @@ When using the `aws-kms` signer type, standard `AWS_*` environment variables are
  - **AWS_SECRET_ACCESS_KEY** (i.e. password)
  - etc.
 
+## Telemetry Style Guide (Tracing + OTEL)
+
+Use `tracing` spans as the default telemetry API.
+
+### Rules
+
+1. Use function/span names as the operation name.
+   - Do not add an `operation = "..."` span field.
+2. Do not attach high-cardinality identifiers to span attributes.
+   - Do not put `txn_id`, `transaction_hash`, or `handle` on spans.
+   - If needed for debugging, log these values in events/log lines.
+3. For async work, instrument futures with `.instrument(...)`.
+   - Do not keep `span.enter()` guards alive across `.await`.
+4. Set OTEL error status on error exits.
+   - Logging an error is not enough for trace error visibility.
+5. Keep span fields low-cardinality and useful for aggregation.
+   - Good examples: `request_id`, counts, booleans, retry bucket, chain id.
+
+### Preferred snippets
+
+```rust
+#[tracing::instrument(skip_all)]
+async fn process_proof(...) -> anyhow::Result<()> {
+    // business logic
+    Ok(())
+}
+```
+
+```rust
+use tracing::Instrument;
+
+let db_insert_span = tracing::info_span!("db_insert", request_id);
+async {
+    sqlx::query("UPDATE ...").execute(pool).await?;
+    Ok::<(), sqlx::Error>(())
+}
+.instrument(db_insert_span.clone())
+.await?;
+```
+
+```rust
+use tracing_opentelemetry::OpenTelemetrySpanExt;
+
+if let Err(err) = do_work().instrument(span.clone()).await {
+    span.context().span().set_status(opentelemetry::trace::Status::error(err.to_string()));
+    return Err(err.into());
+}
+```
+
 
 ## Resources
 

coprocessor/fhevm-engine/Cargo.lock

@@ -1,11 +1,7 @@
 use crate::chain_id::ChainId;
 use crate::utils::to_hex;
 use bigdecimal::num_traits::ToPrimitive;
-use opentelemetry::{
-    global::{BoxedSpan, BoxedTracer, ObjectSafeSpan},
-    trace::{SpanBuilder, Status, TraceContextExt, Tracer, TracerProvider},
-    Context, KeyValue,
-};
+use opentelemetry::{trace::TraceContextExt, trace::TracerProvider, KeyValue};
 use opentelemetry_sdk::{trace::SdkTracerProvider, Resource};
 use prometheus::{register_histogram, Histogram};
 use sqlx::PgConnection;
@@ -14,14 +10,12 @@ use std::{
     num::NonZeroUsize,
     str::FromStr,
     sync::{Arc, LazyLock, OnceLock},
-    time::SystemTime,
 };
 use tokio::sync::RwLock;
-use tracing::{debug, info, warn};
+use tracing::{debug, error, info, warn, Span};
+use tracing_opentelemetry::OpenTelemetrySpanExt;
 use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};
 
-pub const TXN_ID_ATTR_KEY: &str = "txn_id";
-
 /// Calls provider shutdown exactly once when dropped.
 pub struct TracerProviderGuard {
     tracer_provider: Option<SdkTracerProvider>,
@@ -67,18 +61,6 @@ pub(crate) static ZKPROOF_TXN_LATENCY_HISTOGRAM: LazyLock<Histogram> = LazyLock:
     )
 });
 
-pub fn init_otel(
-    service_name: &str,
-) -> Result<Option<TracerProviderGuard>, Box<dyn std::error::Error + Send + Sync + 'static>> {
-    if service_name.is_empty() {
-        return Ok(None);
-    }
-
-    let (_tracer, trace_provider) = setup_otel_with_tracer(service_name, "otlp-layer")?;
-    opentelemetry::global::set_tracer_provider(trace_provider.clone());
-    Ok(Some(TracerProviderGuard::new(trace_provider)))
-}
-
 pub fn init_json_subscriber(
     log_level: tracing::Level,
     service_name: &str,
@@ -115,6 +97,25 @@ pub fn init_json_subscriber(
     Ok(Some(TracerProviderGuard::new(trace_provider)))
 }
 
+/// Initializes tracing with JSON logs and best-effort OTLP export.
+///
+/// Fallback here means "logs-only mode": if OTLP setup fails, we keep
+/// JSON logging enabled and continue execution without an OTLP exporter.
+/// It does not try alternate OTLP endpoints.
+pub fn init_tracing_otel_with_logs_only_fallback(
+    log_level: tracing::Level,
+    service_name: &str,
+    tracer_name: &'static str,
+) -> Option<TracerProviderGuard> {
+    match init_json_subscriber(log_level, service_name, tracer_name) {
+        Ok(guard) => guard,
+        Err(err) => {
+            error!(error = %err, "Failed to setup OTLP");
+            None
+        }
+    }
+}
+
 fn setup_otel_with_tracer(
     service_name: &str,
     tracer_name: &'static str,
@@ -142,119 +143,6 @@ fn setup_otel_with_tracer(
     Ok((tracer, trace_provider))
 }
 
-#[derive(Clone)]
-pub struct OtelTracer {
-    ctx: opentelemetry::Context,
-    tracer: Arc<BoxedTracer>,
-}
-
-impl OtelTracer {
-    pub fn child_span(&self, name: &'static str) -> BoxedSpan {
-        self.tracer.start_with_context(name, &self.ctx)
-    }
-
-    pub fn context(&self) -> &Context {
-        &self.ctx
-    }
-
-    /// Sets attribute to the root span
-    pub fn set_attribute(&self, key: &str, value: String) {
-        self.ctx
-            .span()
-            .set_attribute(KeyValue::new(key.to_owned(), value));
-    }
-
-    /// Consumes and ends the tracer with status Ok
-    pub fn end(self) {
-        self.ctx.span().set_status(Status::Ok);
-        self.ctx.span().end();
-    }
-}
-
-#[derive(Debug, PartialEq)]
-struct Handle(Vec<u8>);
-#[derive(Debug, PartialEq)]
-struct Transaction(Vec<u8>);
-
-pub fn tracer_with_handle(
-    span_name: &'static str,
-    handle: Vec<u8>,
-    transaction_id: &Option<Vec<u8>>,
-) -> OtelTracer {
-    let tracer = opentelemetry::global::tracer(format!("tracer_{}", span_name));
-    let mut span = tracer.start(span_name);
-
-    if !handle.is_empty() {
-        let handle = to_hex(&handle).get(0..10).unwrap_or_default().to_owned();
-
-        span.set_attribute(KeyValue::new("handle", handle));
-    }
-
-    if let Some(transaction_id) = transaction_id {
-        set_txn_id(&mut span, transaction_id);
-    }
-
-    // Add handle and transaction_id to the context
-    // so that they can be retrieved in the application code, e.g. for logging
-    let mut ctx = Context::default().with_span(span);
-    ctx = ctx.with_value(Handle(handle.clone()));
-    ctx = ctx.with_value(Transaction(transaction_id.clone().unwrap_or_default()));
-
-    OtelTracer {
-        ctx,
-        tracer: Arc::new(tracer),
-    }
-}
-
-// Sets the txn_id attribute to the span
-// The txn_id is a shortened version of the transaction_id (first 10 characters of the hex representation)
-pub fn set_txn_id(span: &mut BoxedSpan, transaction_id: &[u8]) {
-    let txn_id_short = to_hex(transaction_id)
-        .get(0..10)
-        .unwrap_or_default()
-        .to_owned();
-
-    span.set_attribute(KeyValue::new(TXN_ID_ATTR_KEY, txn_id_short));
-}
-
-/// Create a new span with start and end time
-pub fn tracer_with_start_time(span_name: &'static str, start_time: SystemTime) -> OtelTracer {
-    let tracer = opentelemetry::global::tracer(span_name);
-    let root_span = tracer.build(SpanBuilder::from_name(span_name).with_start_time(start_time));
-    let ctx = opentelemetry::Context::default().with_span(root_span);
-    OtelTracer {
-        ctx,
-        tracer: Arc::new(tracer),
-    }
-}
-
-pub fn tracer(span_name: &'static str, transaction_id: &Option<Vec<u8>>) -> OtelTracer {
-    tracer_with_handle(span_name, vec![], transaction_id)
-}
-
-pub fn attribute(span: &mut BoxedSpan, key: &str, value: String) {
-    span.set_attribute(KeyValue::new(key.to_owned(), value));
-}
-
-/// Ends span with status Ok
-pub fn end_span(mut span: BoxedSpan) {
-    span.set_status(Status::Ok);
-    span.end();
-}
-
-pub fn end_span_with_timestamp(mut span: BoxedSpan, timestamp: SystemTime) {
-    span.set_status(Status::Ok);
-    span.end_with_timestamp(timestamp);
-}
-
-/// Ends span with status Error with description
-pub fn end_span_with_err(mut span: BoxedSpan, desc: String) {
-    span.set_status(Status::Error {
-        description: desc.into(),
-    });
-    span.end();
-}
-
 #[derive(Clone, Copy, Debug)]
 pub struct MetricsConfig {
     bucket_start: f64,
@@ -326,6 +214,34 @@ pub fn register_histogram(config: Option<&MetricsConfig>, name: &str, desc: &str
         .unwrap_or_else(|_| panic!("Failed to register latency histogram: {}", name))
 }
 
+/// Returns the legacy short-form hex id used by telemetry spans.
+pub fn short_hex_id(value: &[u8]) -> String {
+    to_hex(value).get(0..10).unwrap_or_default().to_owned()
+}
+
+pub fn record_short_hex(span: &Span, field: &'static str, value: &[u8]) {
+    span.record(field, tracing::field::display(short_hex_id(value)));
+}
+
+pub fn record_short_hex_if_some<T: AsRef<[u8]>>(
+    span: &Span,
+    field: &'static str,
+    value: Option<T>,
+) {
+    if let Some(value) = value {
+        record_short_hex(span, field, value.as_ref());
+    }
+}
+
+pub fn set_current_span_error(error: &impl fmt::Display) {
+    tracing::Span::current()
+        .context()
+        .span()
+        .set_status(opentelemetry::trace::Status::Error {
+            description: error.to_string().into(),
+        });
+}
+
 pub(crate) static TXN_METRICS_MANAGER: LazyLock<TransactionMetrics> =
     LazyLock::new(|| TransactionMetrics::new(NonZeroUsize::new(100).unwrap()));
 
@@ -615,10 +531,4 @@ mod tests {
         guard.shutdown_once();
         assert!(guard.tracer_provider.is_none());
     }
-
-    #[test]
-    fn setup_otel_empty_service_name_returns_none() {
-        let otel_guard = init_otel("").unwrap();
-        assert!(otel_guard.is_none());
-    }
 }

coprocessor/fhevm-engine/fhevm-engine-common/src/telemetry.rs

@@ -110,19 +110,11 @@ async fn main() -> anyhow::Result<()> {
 
     let conf = Conf::parse();
 
-    tracing_subscriber::fmt()
-        .json()
-        .with_level(true)
-        .with_max_level(conf.log_level)
-        .init();
-
-    let _otel_guard = match telemetry::init_otel(&conf.service_name) {
-        Ok(otel_guard) => otel_guard,
-        Err(err) => {
-            error!(error = %err, "Failed to setup OTLP");
-            None
-        }
-    };
+    let _otel_guard = telemetry::init_tracing_otel_with_logs_only_fallback(
+        conf.log_level,
+        &conf.service_name,
+        "otlp-layer",
+    );
 
     info!(gateway_url = %conf.gw_url, max_retries = %conf.provider_max_retries,
          retry_interval = ?conf.provider_retry_interval, "Connecting to Gateway");

coprocessor/fhevm-engine/gw-listener/src/bin/gw_listener.rs

@@ -1,14 +1,15 @@
 use clap::Parser;
+use fhevm_engine_common::telemetry;
 
 #[tokio::main]
 async fn main() -> anyhow::Result<()> {
     let args = host_listener::cmd::Args::parse();
 
-    tracing_subscriber::fmt()
-        .json()
-        .with_level(true)
-        .with_max_level(args.log_level)
-        .init();
+    let _otel_guard = telemetry::init_tracing_otel_with_logs_only_fallback(
+        args.log_level,
+        &args.service_name,
+        "otlp-layer",
+    );
 
     host_listener::cmd::main(args).await
 }