add trace contex, el headers and failure replay#70
Merged
han0110 merged 8 commits intoJul 15, 2026
Conversation
Replace the default TraceLayer span with a custom MakeSpan that, when the otel feature is enabled, extracts traceparent/tracestate from request headers via the global propagator and sets the result as the request span's parent. The whole existing explicit-parent chain (request span -> handler -> request_proof -> fetch_witness / prove) now joins the caller's distributed trace. The request span is also promoted from the debug-level tower-http default to info level so it survives info-level filters. Without the otel feature the custom MakeSpan only creates the span, with no OpenTelemetry dependency. Verified by a new otel-gated test using an in-memory span exporter that asserts the exported request span keeps the caller's trace id and is parented under the caller's span id. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Declare an empty job_id field on the per-attempt prove span and pass that span explicitly through zkVMInstance::prove into the zisk cluster backend, which records the id on it as soon as the cluster assigns one, so traces correlate a prove span with the cluster-side job. Recording on the explicitly passed span rather than Span::current() means the id cannot silently land elsewhere if an intermediate span is ever introduced. Also emit an info log with the job id at creation so the correlation is available in plain logs without the otel feature. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add an optional top-level el_headers map (default empty) next to el_endpoint, applied as reqwest default headers when constructing the ElClient, so every EL JSON-RPC request (chain config, blocks, witnesses) carries them, e.g. for authenticated endpoints. Since header values typically carry credentials they are kept out of any Debug output: the parsed HeaderValues are marked sensitive, and Config implements Debug manually so el_headers values format as "<redacted>". Config validation fails at startup on invalid header names or values, and rejects keys that differ only in case (header names are case-insensitive; HashMap iteration order would otherwise decide which duplicate wins). ElClient::new now propagates HTTP-client construction errors instead of panicking, so all startup failures surface as errors. Documented in the README and the example configs; covered by config tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Behind a new otel cargo feature, inject the current tracing span's OpenTelemetry context (traceparent/tracestate) into the headers of every outbound request (request_proof, subscribe_proof_events, get_proof, verify_proof) via the global propagator, using a hand-rolled HeaderInjector mirroring ere_server_client's OtelPropagation middleware and the server crate's HeaderExtractor. For the SSE subscription the context is captured eagerly at call time, not at first poll. Without the feature the helper returns an empty header map, so default builds change no behavior and pull in no OpenTelemetry dependencies. The crate docs call out that the calling application must install a global text-map propagator (the OpenTelemetry default is a no-op), or no traceparent header is emitted even with the feature enabled. This completes the propagation chain for external callers such as proofessoor: caller span -> zkboost-client traceparent -> zkboost server request span -> ere zkVM server. Verified by an otel-gated integration test that serves requests on a local listener and asserts both a unary request and the SSE subscribe arrive with a traceparent carrying the active span's trace id. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Set otel.kind=server and basic HTTP semantic-convention attributes (http.request.method, url.path, url.query) on the request span. They are recorded as OTLP-only attributes, not tracing fields, so the log format of default-feature builds is unchanged. Also move the request-span trace-context test into its own integration-test binary (tests/otel_request_span.rs): it installs a process-global tracing subscriber so spans created on the server's tasks are observed, and sharing a process with parallel tests both raced on the global subscriber and poisoned the global callsite interest cache, which previously required a warm-up/retry workaround. The relocated test additionally asserts the span kind and HTTP attributes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Previously only completed proofs were cached and replayed on subscribe; a subscriber that missed a live proof_failure broadcast could never learn the terminal outcome of a request. Terminal failures are now cached in a bounded LRU (same size as the completed-proof cache) and replayed to per-root SSE subscribers exactly like completions, with unchanged event shapes. A later successful proof for the same (root, proof_type) evicts the stale failure entry so both outcomes are never replayed. Event delivery on the subscribe endpoint is at-least-once with latest-wins semantics: a replayed stale failure can still precede the completion of a retry that has already succeeded, and subscribers must treat the most recent terminal event per (root, proof_type) as authoritative. This is documented on the handler and in the README. Tests: subscribe-after-failure receives the replayed failure (handler unit test + end-to-end re-subscribe assertions for proving error and timeout), LRU bound eviction, cached-vs-broadcast event equality, and completion evicting a stale failure. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
chetanyb
changed the title
han0110
reviewed
Jul 15, 2026
han0110
left a comment
han0110
left a comment
Collaborator
There was a problem hiding this comment.
Mostly LGTM! Left some questions
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
job_id.Failure replay uses a bounded, process-local LRU with latest-wins semantics. A later success evicts a stale failure.
Validation