feat(dm): durable send stage timers (#336 phase 1)

[dirvine]

Summary

Phase 1 of #336 only: measure the first durable DM to a new peer. This does not cut latency.

A durable gossip-inbox send now records three named stages that partition daemon-side wall time:

1. strict_gate_ms — advert / capability store (ADR 0030 §2 refresh)
2. publish_ms — inbox / topic fan-out
3. ack_wait_ms — sender waiter vs receiver ACK publish-complete

budget_stage names which of the three consumed the most time, so a slow send can say which stage ate the budget. Until that exists, no "we warmed X" merge.

Exported on:

• the existing send-timeout 504 body (error stays timeout, detail stays the existing Display string)
• GET /diagnostics/dm as last_durable_send (sender) and last_ack_publish_ms (receiver ACK publish)

Out of scope (deliberately not in this PR)

• Phase 2 latency cuts
• Treating feat(dm): bounded background publisher for durable ACKs (closes #335) #338 as closing cold-start latency: first durable DM to a new peer costs ~17.5s (gossip cold success path) #336
• Flipping prefer_raw_quic_if_connected on for durable sends
• Reopening durable ACK publication is inline-awaited in the inbox loop (no timeout, no bounded worker) — port hedged ACK publisher stack post-0.38 #335's ACK contract
• Client 30s timeouts, daemon 8s×2 + 8s backoff budget
• Writing 17.5s into api-reference as an SLA
• HTTP status or ack-semantics contract changes
• upsert_discovered_machine: unbounded address accumulation + 5s-per-dead-candidate reconnect cost #308, [groups] Group public message send returns 200 + msg_id while publishing to zero peers — silent black-hole needs surfacing #296, Group message delivery takes ~44s when direct-unicast path fails — 24s timeout + retry before gossip fallback lands #310, [config] misplaced data_dir under [gossip] silently falls back to default — multi-daemon boxes collide on history.db #281

Validation

• cargo fmt --all
• cargo clippy --locked --all-features --all-targets -- -D warnings
• cargo check --workspace --all-targets
• New unit tests (stage naming, 504 body, diagnostics snapshot, tracker partition)
• cargo test --locked --all-features --test durable_send_stage_timers (two-daemon loopback first send: stages sum to wall)

Coverage

• Current line coverage: not collected in this revision
• Delta vs main: n/a
• Coverage workstream, if applicable: n/a
• Exclusions added: none

Test Quality Checklist

• Each new test has a why-named name that states the invariant or regression.
• No new test is a tautology against the implementation.
• Each new test checks one business invariant.
• Assertion failures include enough context to diagnose the broken invariant.
• Coverage delta is reported from CI or just coverage-summary.
• Any coverage exclusion has a coverage-skip: comment and a matching register entry.

  

Greptile Summary

The PR adds sender-side durable-DM stage timing and receiver ACK-publish timing to identify where first-send latency is spent.

• Tracks strict capability gating, gossip publishing, and ACK-wait time across retries.
• Exposes daemon-wide timing snapshots through timeout responses and /diagnostics/dm.
• Adds unit and two-daemon integration coverage for stage partitioning.

Confidence Score: 4/5

The timeout-stage correlation defect should be fixed before merging because concurrent sends can return another request's diagnostics.

Durable sends overwrite one daemon-wide timing slot, while a timed-out handler reads that slot later without request correlation, allowing an overlapping send to replace the measurements attached to the 504 response.

Files Needing Attention: src/server/routes/direct.rs, src/direct.rs, src/lib.rs

Important Files Changed

Filename	Overview
src/dm_send.rs	Adds a drop-guarded stage tracker that partitions gossip publication, ACK waiting, and retry backoff across all return paths.
src/lib.rs	Measures strict capability gating and total durable-send elapsed time, then records each result in daemon-wide diagnostics.
src/direct.rs	Adds last-write-wins sender and receiver timing snapshots shared by all requests.
src/server/routes/direct.rs	Adds stage fields to timeout responses, but retrieves them from an uncorrelated shared snapshot that concurrent sends can overwrite.
src/dm_inbox.rs	Measures completion time for each bounded, hedged durable-ACK publication job.
src/server/routes/network.rs	Exposes the latest sender-stage and receiver ACK-publish measurements through /diagnostics/dm.
tests/durable_send_stage_timers.rs	Validates single-send timer partitioning and API projection but does not exercise concurrent timeout attribution.

Sequence Diagram

sequenceDiagram
  participant Client
  participant API as /direct/send
  participant Agent
  participant Gossip
  participant Peer
  Client->>API: durable send
  API->>Agent: send_direct_with_config
  Agent->>Agent: measure strict_gate_ms
  Agent->>Gossip: publish inbox envelope
  Note over Agent,Gossip: measure publish_ms
  Gossip->>Peer: durable DM
  Peer->>Gossip: publish durable ACK
  Gossip-->>Agent: resolve ACK waiter
  Note over Agent,Gossip: measure ack_wait_ms
  Agent->>Agent: overwrite last_durable_send
  Agent-->>API: receipt or timeout
  API->>Agent: read last_durable_send on timeout
  API-->>Client: response with stage timers

Loading

Prompt To Fix All With AI

### Issue 1
src/server/routes/direct.rs:501-509
**Timeout stages lose request correlation**

If two durable `/direct/send` requests overlap, each send overwrites the same daemon-wide `last_durable_send` slot before the timeout handler reads it, causing a 504 response to report another request's stage timings and `budget_stage`.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

_{Reviews (1): Last reviewed commit: "feat(dm): add durable send stage timers ..." | Re-trigger Greptile}

Greptile also left 1 inline comment on this PR.

Context used:

• Knowledge Base — Direct Messaging: Send, Capability Gate, Inbox, and Forward
• Knowledge Base — Server API: HTTP/WebSocket/SSE surface and agent signing

[greptile-apps]

Timeout stages lose request correlation

If two durable /direct/send requests overlap, each send overwrites the same daemon-wide last_durable_send slot before the timeout handler reads it, causing a 504 response to report another request's stage timings and budget_stage.

Knowledge Base Used:

• Direct Messaging: Send, Capability Gate, Inbox, and Forward
• Server API: HTTP/WebSocket/SSE surface and agent signing

Prompt To Fix With AI

This is a comment left during a code review.
Path: src/server/routes/direct.rs
Line: 501-509

Comment:
**Timeout stages lose request correlation**

If two durable `/direct/send` requests overlap, each send overwrites the same daemon-wide `last_durable_send` slot before the timeout handler reads it, causing a 504 response to report another request's stage timings and `budget_stage`.

**Knowledge Base Used:**
- [Direct Messaging: Send, Capability Gate, Inbox, and Forward](https://app.greptile.com/saorsa-labs/-/custom-context/knowledge-base/saorsa-labs/x0x/-/docs/dm-messaging.md)
- [Server API: HTTP/WebSocket/SSE surface and agent signing](https://app.greptile.com/saorsa-labs/-/custom-context/knowledge-base/saorsa-labs/x0x/-/docs/server-api.md)

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.