Skip to content

wire-protocol.md

Latest commit

 

History

History
46 lines (36 loc) · 9.83 KB

wire-protocol.md

File metadata and controls

46 lines (36 loc) · 9.83 KB
Raw

Wire protocols: daemon ↔ clients, daemon ↔ workers

Verified against code 2026-08-15.

AfterRay has three separate JSON protocols: the versioned control socket between afterrayd and its clients (CLI, SwiftUI app), the one-shot worker protocol (OCR/ASR/embedding), and the persistent MLX worker protocol (local LLM). Each has its own version constant; bump both sides of whichever you touch. Control protocol 21 adds raw-evidence retention, GOP quality aging, and the framed measured-quality preview.

1. Control socket: afterrayd ↔ CLI / SwiftUI app

  • Single source of truth: crates/afterray-protocol. Request is tagged snake_case; responses use {protocol_version, ok, data?, error?}; PROTOCOL_VERSION = 21. Unprivileged peers (CLI/agents) are gated in the daemon: Query always, Evidence only while cli_evidence_until_ms is in the future, Privileged (writes/ask/chat) never. The app is identified by socket audit token + a valid dev.afterray.app signature whose Team ID matches, or whose cdhash matches the AfterRay process that spawned the daemon. Identifier-only ad-hoc signatures are not enough.
  • Framing: one request = one JSON object + \n over a Unix socket. Three response shapes:
    • single JSON line — the default, served by dispatch (crates/afterrayd/src/main.rs:625);
    • artifact reads (ReadArtifact / ReadGopSegment / ReadGopFrame / ReadThumbnail) — a JSON header line (ArtifactMeta) followed by exactly byte_length raw bytes;
    • ChatStream — NDJSON ChatStreamEvent lines until done/error.
    • Binary/streaming requests are intercepted in the daemon's handle loop (main.rs:529) before dispatch; dispatch fails them if reached. New binary or streaming requests must follow that split.
  • Agent screenshot citations add no wire shape: a complete ![label](afterray://moment/ID) (standalone, indented, list-prefixed, or inline) is parsed by the Swift chat surface. First paint uses ReadThumbnail (filmstrip JPEG, typically 360px). The card then loads MomentGet for captured_at_ms and upgrades the picture via ReadArtifact(image_artifact_id) or ReadGopFrame Exact — not the cached thumbnail, whose max_edge is ignored on a cache hit. General Markdown image URLs are never fetched; deleted or invalid moments degrade to a clickable text citation.
  • usage events carry tokenizer-accurate prompt_tokens / completion_tokens / generation_ms when the runtime reports them (Ollama eval_count, OpenAI usage, MLX GenerateCompletionInfo). Missing fields stay 0. The context ring is prompt_tokens / window_tokens; tok/s is completion_tokens / generation_ms. Do not invent these from character counts.
  • Swift mirror: swift/AfterRayRecall/Sources/DaemonClient.swift — UnixSocketDaemonClient, a hand-declared WireRequest with snake_case CodingKeys, and protocolVersion = 18 enforced on every response. Bump Rust and Swift together — there is no negotiation; a mismatch fails every request with protocolMismatch. List reads (timeline_range, timeline_list, timeline_since) are a lean index: no concatenated OCR, transcripts, or subtitle cues. They carry one exact audio segment id/artifact/start/end; only moment_get adds that segment's transcript and its ordered relative-time cues. OCR stays on moment_get / evidence_ocr. The overlay fetches one local day at a time and slides that window as the playhead approaches an edge.
  • Evolution rules: additive-only; new optional fields use #[serde(default, skip_serializing_if = "Option::is_none")]. Never rename variants/fields — the *_wire_shape_is_stable tests in protocol lib.rs pin exact JSON bytes. For enums persisted in settings, follow LlmProvider: lenient custom Deserialize mapping retired/unknown labels to the default, strict serialization. Mirror every new field in Swift's WireRequest and add a wire-shape test (Swift side: DaemonWireTests / ChatWireTests).
  • Socket path resolution lives only in crates/afterray-protocol/src/socket.rs (default_socket_path, line 22): AFTERRAY_SOCKET env → <checkout>/.afterray-dev/afterray.sock (only when the executable sits under target/{debug,release}) → ~/Library/Application Support/AfterRay/afterray.sock. Daemon, CLI, and app must all resolve through this — they used to diverge.
  • Security: the daemon binds the socket 0600 inside a 0700 directory, rejects symlink/non-socket/foreign-owned paths, and re-checks the peer uid per connection (bind_control_socket, afterrayd main.rs:57; peer check main.rs:251). Artifact bytes travel the socket already decrypted — the filesystem boundary is the entire access control. ArtifactPayload zeroizes its bytes on Drop (protocol lib.rs:761).
  • slot_summary_export {at_ms} returns actual slot bounds/state, raw persisted schema version, parsed P2, visible facts and generation metadata. Its store query intentionally excludes OCR, AX, evidence, prompts, tool results and raw model completions.
  • Summary schema 1 exports the original title, bullets, artifacts, category and confidence shape; schema 2 exports the structured description/thread shape. Both remain valid across the protocol 12 upgrade.
  • Empty threads[].moment_ids is omitted by Rust; Swift readers must decode an absent key as []. One missing citation list must never reject the entire day-summary payload.
  • A summary row carries whichever card shape it was written in: v1 bullets, v2 threads/entities/decisions, or v3 details (one Markdown document, protocol 15). Every one of them is optional on the wire and schema_version on the export says which is authoritative — a reader that guesses from nullness reads the newest card as the oldest shape.
  • Status.host_build echoes AFTERRAY_HOST_BUILD (protocol lib.rs:318) so the app can detect and restart a stale daemon after an in-place update — a separate concern from protocol_version.

2. One-shot worker protocol (OCR / ASR / alignment / embedding)

  • crates/afterray-models/src/process.rs: WORKER_PROTOCOL_VERSION = 2 (line 7). One child process per inference: exactly one JSON request on stdin, exactly one JSON response on stdout, stderr = logs only. 300s timeout, 16 MiB stdout cap (process.rs:57-58). The response must echo the protocol version and its output capability must match the request.
  • Speakers: the Rust afterray-model-worker (crates/afterray-infer/src/bin/afterray-model-worker.rs — ASR + alignment + embedding; rejects OCR and LLM) and the Swift apps/AfterRayNativeModelWorker (macOS Vision OCR; reports errors as {error, retryable} on stdout and still exits 0).
  • Retry contract: retryable: true maps to retryable AdapterError::Process; false maps to MissingModel. Only Process/Io/Timeout errors retry (AdapterError::retryable).

3. Persistent MLX worker protocol (local LLM and ASR)

  • crates/afterray-models/src/persistent_mlx.rs: MLX_WORKER_PROTOCOL_VERSION = 3. A long-lived child speaks newline-delimited JSON both ways. VLM requests are load/generate/cancel; ASR requests are load/asr_generate. Responses are ready/delta/final/cancelled/error, each echoing request_id. final means successful completion: for ASR its required text may be ""; the daemon then discards that segment and its encrypted audio artifact because there is no recognised speech. error means failure and carries retryable; error text is diagnostic only, never a scheduling signal. Each VLM generate carries a complete, stateless prompt; the protocol has no cross-request KV/session-reuse field. Load timeout 180s, generate 300s, restart backoff 1s. verify_model re-checks the pinned manifest + .afterray-ready.json marker before every spawn.
  • Swift sides: swift/AfterRayMlxVlmWorker/Sources/WorkerCore.swift and apps/AfterRayMlxAsrWorker/Sources/main.swift use protocol v3. The VLM worker is single-flight: one generate at a time; every generate gets a fresh ChatSession; cancel is acknowledged only after the MLX task actually stops. The ASR worker serves one serial request at a time and its synchronous generation is cancelled by process termination. The model container stays resident only while the worker is warm; the daemon ends the process after 120s without a request. Revision pins qwen35_4BRevision/qwen35_9BRevision must stay equal to QWEN35_4B_MLX_REVISION/QWEN35_9B_MLX_REVISION in crates/afterray-models/src/catalog.rs (verified equal today).
  • Remote LLM alternative: LlmRouterAdapter (crates/afterray-models/src/remote.rs:127) routes to Ollama/OpenAI-compatible HTTP only through check_origin (remote.rs:484 — https, or http only to loopback); reqwest clients are built with redirects disabled so prompts and API keys can't leak to a redirect target.

Watch out

  • stdout of every worker is protocol-only — a stray print kills the job as invalid output (there is a regression test for this). Logs go to stderr (WorkerLog in Swift).
  • $TMPDIR/afterray-v0.sock is the retired socket default. Never fall back to $TMPDIR (world-writable, pre-bindable), and never key dev-socket detection off the working directory — only off the executable path. The Swift client's last-resort $TMPDIR fallback is dead in practice (the app always passes a path explicitly); don't "fix" Rust to match it.
  • Dead-but-present wire surface: Request::FavoriteSet (daemon replies "favorites are disabled", main.rs:676), PackStatus.keep_stills (always false), LlmProvider labels builtin/local (retired GGUF backends, kept decode-only).
  • PackStatus also carries ready_frames and one_frame_segments (additive defaults): mean frames/GOP is ready_frames / ready_segments. A rising one_frame_segments share means the per-resolution fold regressed.
  • Fixture JSON in daemon tests shows "protocol_version": 1 — that is test data, not the real version.
  • afterray download bypasses the daemon and uses afterray-models directly — intentional, not a bug.