Merge pull request #13 from prisma/codex/otel-traces-observability

[codex] Add OTel request observability

Author: sorenbs

README.md

@@ -119,6 +119,7 @@ Current built-ins:
 - `evlog`
 - `generic`
 - `metrics`
+- `otel-traces`
 - `state-protocol`
 
 Planned next built-ins:
@@ -189,6 +190,25 @@ It means:
 The internal `__stream_metrics__` stream is created with this profile
 automatically.
 
+### `otel-traces`
+
+`otel-traces` is the built-in profile for OpenTelemetry trace spans.
+
+It means:
+
+- the stream content type must be `application/json`
+- JSON appends are normalized into the canonical span envelope
+- OTLP trace exports are accepted through `POST /v1/traces` and
+  `POST /v1/stream/{name}/_otlp/v1/traces`
+- installing the profile also installs the canonical trace schema/search and
+  default rollups
+- the canonical routing key is `traceId`
+
+See [docs/profile-otel-traces.md](./docs/profile-otel-traces.md) for the
+profile and OTLP receiver contract, and
+[docs/request-observability.md](./docs/request-observability.md) for
+cross-stream lookup over `evlog` events and `otel-traces` spans.
+
 ## Profile Versus Schema
 
 What belongs in a profile:
@@ -407,11 +427,14 @@ Not implemented today:
 
 The supported behavior is:
 
-- use `/_profile` to choose `generic`, `state-protocol`, or `evlog`
+- use `/_profile` to choose a built-in profile, including `generic`, `evlog`,
+  `metrics`, `otel-traces`, and `state-protocol`
 - use `/_schema` only for schema validation, routing-key config, and schema
   evolution
 - use `/touch/*` only on `state-protocol` streams with touch enabled
 - use normal JSON appends on `evlog` streams to store canonical evlog events
+- use OTLP trace endpoints only on `otel-traces` streams, or on `/v1/traces`
+  when `DS_OTLP_TRACES_STREAM` is configured
 
 Legacy compatibility branches are intentionally not part of the supported
 surface.

docs/architecture.md

@@ -32,14 +32,17 @@ Implemented built-ins today:
 - `evlog`
 - `generic`
 - `metrics`
+- `otel-traces`
 - `state-protocol`
 
 `generic` adds no canonical payload envelope and leaves schema management to the
 user. `evlog` owns canonical wide-event normalization, redaction, and its
 default schema/search/rollup registry on JSON append. `metrics` owns canonical
 metrics interval normalization, its default schema/search/rollup registry, and
-the metrics-block companion family. `state-protocol` owns the live `/touch/*`
-surface and its touch configuration.
+the metrics-block companion family. `otel-traces` owns canonical span
+normalization, OTLP trace export decoding, redaction, backend-side trace limits,
+and its default schema/search/rollup registry. `state-protocol` owns the live
+`/touch/*` surface and its touch configuration.
 
 See [stream-profiles.md](./stream-profiles.md) for the normative model.
 
@@ -51,6 +54,8 @@ See [stream-profiles.md](./stream-profiles.md) for the normative model.
 - Implements long-poll reads without busy loops.
 - Resolves the stream profile definition before handling profile-owned
   metadata or routes.
+- Uses profile capabilities for OTLP trace ingestion and correlation timeline
+  conversion instead of hard-coding profile branches in the core route path.
 - Admits ingest, read, and search work through bounded in-process concurrency
   gates instead of a direct memory-based reject path.
 
@@ -151,6 +156,20 @@ Today, `metrics` uses the same model to own:
 - bundled per-segment `PSCIX2` `.cix` search companions for metrics-serving
   state
 
+Today, `otel-traces` uses the same model to own:
+
+- canonical OpenTelemetry span normalization on JSON append
+- OTLP JSON/protobuf trace export decoding on `POST /v1/traces` and
+  `POST /v1/stream/{name}/_otlp/v1/traces`
+- pre-append redaction and backend-side attribute/event/link limits
+- routing-key defaults from `traceId`
+- default schema-owned `search` and `search.rollups` installation
+
+The cross-stream request observability API is a query layer over `evlog` and
+`otel-traces` streams. It uses stream search results and profile correlation
+capabilities to build summaries, trace trees, service edges, and timelines; it
+does not create a separate mutable observability store.
+
 ## Control-Plane Metadata
 
 Per stream, SQLite stores:

docs/durable-streams-spec.md

@@ -59,12 +59,19 @@ implementation.
 - `GET  /v1/stream/{name}/_routing_keys` list routing keys alphabetically
 - `GET  /v1/stream/{name}/_index_status` get per-stream index status
 - `GET  /v1/stream/{name}/_details` get combined stream details
+- `POST /v1/stream/{name}/_otlp/v1/traces` ingest OTLP traces into an
+  `otel-traces` stream
 
-### 2.5 Streams collection
+### 2.5 Observability resources
+
+- `POST /v1/traces` ingest OTLP traces into `DS_OTLP_TRACES_STREAM`
+- `POST /v1/observe/request` correlate request events and trace spans
+
+### 2.6 Streams collection
 
 - `GET /v1/streams` list streams
 
-### 2.6 Server inspection
+### 2.7 Server inspection
 
 - `GET /v1/server/_details` get server-scoped configured limits and live runtime state
 
@@ -211,6 +218,132 @@ Optional fields:
 - `group_by`
 - `measures`
 
+### 4.4 OTLP trace ingestion
+
+`POST /v1/traces` accepts OTLP trace export requests and writes accepted spans
+to the stream named by `DS_OTLP_TRACES_STREAM`.
+
+Rules:
+
+- `DS_OTLP_TRACES_STREAM` must be configured, otherwise the endpoint returns
+  `400`.
+- If the target stream does not exist and `DS_OTLP_AUTO_CREATE=true`, the
+  server creates an `application/json` stream, installs the `otel-traces`
+  profile, uploads the profile-owned schema registry, publishes a manifest, and
+  then appends accepted spans.
+- If the target stream does not exist and auto-create is not enabled, the
+  endpoint returns `404`.
+- The target stream must have the `otel-traces` profile.
+
+`POST /v1/stream/{name}/_otlp/v1/traces` accepts the same OTLP payloads for an
+explicit stream. The stream must already exist and have the `otel-traces`
+profile.
+
+Supported request content types:
+
+- `application/x-protobuf`
+- `application/json`
+
+Supported content encodings:
+
+- no encoding / `identity`
+- `gzip`
+
+Successful full acceptance returns HTTP `200` with an empty OTLP
+`ExportTraceServiceResponse` for protobuf or `{}` for JSON. Partial acceptance
+also returns HTTP `200` and includes OTLP partial-success information with the
+number of rejected spans and an error message. Clients must not retry spans
+rejected by a partial-success response.
+
+Malformed payloads and requests that exceed resource-span or scope-span limits
+return `400`. Compressed or decoded OTLP bodies that exceed the configured byte
+limits return `413`. Unsupported content types or content encodings return
+`415`. If a decodable request exceeds the configured span-count limit, the
+server accepts the first spans up to the limit and returns HTTP `200` with OTLP
+partial-success information for the rejected overflow. Accepted spans are
+appended as canonical JSON span records using `traceId` as the routing key.
+
+### 4.5 Request observability
+
+`POST /v1/observe/request` correlates an event stream and a trace stream at
+query time. It does not append data and does not create a new stream profile.
+
+Request body:
+
+```json
+{
+  "streams": {
+    "events": "app-events",
+    "traces": "app-traces"
+  },
+  "lookup": {
+    "requestId": "req_123"
+  },
+  "time": {
+    "from": "2026-03-27T00:00:00.000Z",
+    "to": "2026-03-28T00:00:00.000Z",
+    "paddingMs": 5000
+  },
+  "include": {
+    "events": true,
+    "trace": true,
+    "timeline": true,
+    "raw": false
+  },
+  "limits": {
+    "events": 100,
+    "spans": 5000
+  }
+}
+```
+
+`lookup` must contain exactly one of `requestId`, `traceId`, or `spanId`.
+`streams.events` is required when `include.events=true`; `streams.traces` is
+required when `include.trace=true`.
+The supported request-observability pairing is `streams.events` with profile
+`evlog` and `streams.traces` with profile `otel-traces`.
+
+The endpoint uses the configured `_search` registries for the referenced
+streams. Event and trace streams must expose the profile correlation capability.
+`include.raw` defaults to `false`. With `raw=false`, `evlog.primary`,
+`evlog.matches[].source`, and `trace.spans[]` contain compact normalized records
+that keep IDs, timestamps, service/request fields, status/error fields, and
+safe request/operation summaries while omitting raw context, attributes,
+resources, span events, links, statements, URLs, stack traces, redaction
+metadata, and identity internals. Timeline items omit `data`. With `raw=true`,
+those response fields include the full profile-normalized source records and
+timeline source data.
+The response contains:
+
+- `lookup`
+- `summary`
+- `evlog`
+- `trace`
+- `timeline`
+- `coverage`
+
+The trace response deduplicates returned spans by `traceId:spanId` for the
+tree, service map, errors, and critical path. Duplicate span records remain in
+the underlying append-only stream.
+
+`trace.rootSpanId` is selected from all returned root candidates by preferring
+likely request roots: no parent, server kind, HTTP fields, request ID, and then
+duration. Other roots remain in `trace.tree`. `trace.criticalPath` is a
+best-effort interval-aware latency path from the selected root when one exists.
+
+`coverage.events` and `coverage.traces` de-duplicate `hits`, `unique_hits`, and
+`total.value` by stream and offset across overlapping lookup searches.
+`query_count` and `batch_count` report the number of underlying `_search`
+batches used. `total.relation` is `gte` when limits, timeouts, incomplete
+coverage, or underlying lower-bound totals prevent an exact unique total. Each
+coverage object also includes `queries`, preserving per-query diagnostics such
+as `q`, returned `hits`, backend `total`, page count, timeout state, and limit
+state.
+
+The endpoint returns `400` for invalid request bodies, unsupported profile
+combinations, or streams without search configuration. Missing streams return
+`404`.
+
 ---
 
 ## 5. Offsets

docs/index.md

@@ -25,8 +25,12 @@ and tooling.
 - [stream-profiles.md](./stream-profiles.md) — stream/profile/schema model and profile subresource
 - [profile-generic.md](./profile-generic.md) — reference for the baseline `generic` profile
 - [profile-metrics.md](./profile-metrics.md) — reference for the built-in `metrics` profile
+- [profile-otel-traces.md](./profile-otel-traces.md) — reference for the built-in
+  `otel-traces` profile and OTLP trace ingestion
 - [profile-state-protocol.md](./profile-state-protocol.md) — reference for the `state-protocol` profile
 - [profile-evlog.md](./profile-evlog.md) — design and reference for the `evlog` profile
+- [request-observability.md](./request-observability.md) — cross-stream request
+  lookup that correlates `evlog` events and `otel-traces` spans
 - [schemas.md](./schemas.md) — schema registry and lens behavior
 - [durable-lens-v1-schema.md](./durable-lens-v1-schema.md) — reference schema for `durable.lens/v1`
 - [sqlite-schema.md](./sqlite-schema.md) — SQLite schema, invariants, and migration expectations

docs/overview.md

@@ -15,11 +15,19 @@ Every stream has a profile.
   validation.
 - `metrics` is the built-in metrics profile for canonical interval summaries,
   default search/rollups, and object-store-native metrics companions.
+- `otel-traces` is the built-in OpenTelemetry trace profile for one canonical
+  JSON span per record, OTLP trace ingestion, trace search/rollups, and request
+  correlation with `evlog`.
 - `state-protocol` is the built-in live/touch profile for JSON State Protocol
   streams.
 - Profiles define stream semantics; schemas define payload shape.
 
 See [stream-profiles.md](./stream-profiles.md).
+See [profile-otel-traces.md](./profile-otel-traces.md) and
+[request-observability.md](./request-observability.md) for trace ingestion and
+cross-stream request lookup. UIs should use the explicit
+`observability.request` descriptor from `GET /v1/streams` or
+`GET /v1/stream/{name}/_details` to pair `evlog` and `otel-traces` streams.
 
 This repository currently contains two server modes:
 
@@ -180,6 +188,13 @@ Optional flags:
 - `--bootstrap-from-r2`
 - `--auto-tune[=MB]`
 
+Optional OTLP trace receiver configuration:
+
+- `DS_OTLP_TRACES_STREAM=<stream>` enables the default `POST /v1/traces`
+  receiver target
+- `DS_OTLP_AUTO_CREATE=true` lets `/v1/traces` create and profile that stream
+  as `otel-traces` before accepting spans
+
 ### Object Store Configuration
 
 Local MockR2:

docs/profile-evlog.md

@@ -19,6 +19,9 @@ The v1 profile:
 - support request-centric lookup through the existing routing-key path
 
 The v1 profile does not introduce a separate observability storage engine.
+It also does not store OpenTelemetry span graphs; spans belong in
+[`otel-traces`](./profile-otel-traces.md) streams and are correlated at query
+time.
 
 ## Stream Contract
 
@@ -31,8 +34,58 @@ The v1 profile does not introduce a separate observability storage engine.
   default `search` and `search.rollups` config
 - the profile provides a default routing key from `requestId`, with `traceId`
   fallback
+- optional correlation settings can define request/trace field aliases and
+  `traceparent` parsing for better joins with `otel-traces`
 - reads continue to use the normal durable stream APIs
 
+Supported profile shape:
+
+```json
+{
+  "kind": "evlog",
+  "redactKeys": ["sessiontoken"],
+  "correlation": {
+    "requestIdFields": ["requestId", "context.requestId"],
+    "traceContextFields": [
+      "traceId",
+      "spanId",
+      "traceContext.traceId",
+      "traceContext.spanId"
+    ],
+    "parseTraceparent": true
+  },
+  "observability": {
+    "request": {
+      "tracesStream": "app-traces"
+    }
+  }
+}
+```
+
+`correlation` only affects how the evlog canonical envelope derives
+`requestId`, `traceId`, and `spanId`; it does not make evlog accept spans.
+When `parseTraceparent` is not false, the profile reads W3C `traceparent` from
+`traceparent`, `traceContext.traceparent`, `context.traceparent`, or
+`headers.traceparent` if explicit trace fields are absent.
+
+`observability.request.tracesStream` declares the explicit `otel-traces`
+counterpart for request-observability clients. When it is present,
+`GET /v1/streams` and `GET /v1/stream/{name}/_details` expose:
+
+```json
+{
+  "observability": {
+    "request": {
+      "events_stream": "app-events",
+      "traces_stream": "app-traces"
+    }
+  }
+}
+```
+
+Clients must use this descriptor instead of guessing the trace stream from
+other stream names or profiles.
+
 ## Canonical Envelope
 
 Each stored event should use this stable top-level shape:
@@ -196,6 +249,7 @@ Current evlog query surfaces:
 - `POST /v1/stream/{name}/_search`
 - `GET /v1/stream/{name}/_search?q=...`
 - `POST /v1/stream/{name}/_aggregate`
+- `POST /v1/observe/request` when paired with an `otel-traces` stream
 
 ## UI Integration
 
@@ -206,7 +260,9 @@ record/detail surface.
 Recommended integration flow:
 
 1. Create the stream with `application/json`.
-2. Install the `evlog` profile with `POST /v1/stream/{name}/_profile`.
+2. Install the `evlog` profile with `POST /v1/stream/{name}/_profile`. Include
+   `observability.request.tracesStream` when this stream has a known
+   `otel-traces` counterpart.
 3. Read `GET /v1/stream/{name}/_details` when the UI needs the combined
    stream/profile/schema/index descriptor.
 4. Read `GET /v1/stream/{name}/_index_status` for dedicated indexing progress