feat(server,types): stage-timing observability — queue-wait metric, self-describing completions, identified spans

[chetanyb]

Problem

Per-stage timing of a proof request is only partially observable:

• zkboost_prove_duration_seconds starts after a worker dequeues the job — time waiting in the worker channel between dispatch and dequeue appears in no metric. Under sustained load that wait dominates: we observed traces where request_proof spanned 5m16s containing only ~6.6s of witness fetch and ~21s of proving, the rest being invisible queue wait.
• Consumers wanting per-request stage attribution must either poll /dashboard/state (bounded retention, lost on restart) or subtract span timestamps in a trace backend — both lossy reconstructions of numbers the server already holds at completion time.
• The stage spans (request_proof, fetch_witness, prove/*) carry no identifying attributes, so span rows in trace UIs can't be related to a block without walking the parent chain.

Changes

Commit 1 — metrics:

1. New zkboost_queue_wait_duration_seconds{proof_type} — an Instant stamped into WorkerInput at dispatch, observed at dequeue. Long-tail buckets (0.05s … 900s): backlogs of minutes are exactly what this metric exists to expose.
2. zkboost_witness_fetch_duration_seconds bucket fix — the default buckets end at 10.0, hiding successful 10–12s fetches in +Inf despite the slot-aligned witness timeout; now the same 0.5s-step bounds to 12.0 as the prove histogram.

Commit 2 — self-describing completions + span identity:

3. ProofComplete gains optional witness_ms / queue_wait_ms / prove_ms (Option<u64> + #[serde(default)]). The three stages exactly partition the request lifecycle: admission → witness available → dequeue → done. Absence means unknown, not zero — old producers and replayed cache entries (which hold only proof bytes) send None. Producers and consumers can upgrade in either order; serde-compat tests cover both directions plus a populated round trip.
4. Identity attributes on stage spans: new_payload_request_root on request_proof and prove; block_hash on fetch_witness (the witness service is keyed by execution block hash and serves all requests for that block, so the hash is its identity).

Commit 3 — the same fields on failures:

5. ProofFailure gains the same optional trio, with one semantic addition: on failures, None also means the stage never ran, so the populated/absent shape locates how far a request got before dying — a witness-timeout failure carries only witness_ms; a proving-timeout carries all three with prove_ms = the exhausted budget; an admission-time rejection carries none. Timings are threaded to each failure site via a small all-None-default StageTimings carrier. Replayed failures keep their timings for free (the failure cache stores the whole event). Without this, stage data would exist only for successes — while failures are the primary subjects of stage-attribution analysis.

Notes

• No behavior change — measurement and event enrichment only; existing consumers are unaffected (unknown JSON keys are ignored, no deny_unknown_fields anywhere on this path).
• cargo fmt / clippy --all-targets --locked -- -D warnings / cargo test --locked (80 passed) all clean.
• With this, witness + queue_wait + prove ≈ request_proof closes arithmetically, per-request stage data arrives with completion and failure events instead of being polled, and every stage span self-identifies in trace UIs.