Add FedifySpanExporter for distributed trace storage

AGENTS.md

@@ -57,6 +57,7 @@ The repository is organized as a monorepo with the following packages:
     -  *src/compat/*: Compatibility layer
     -  *src/federation/*: Core federation functionality
     -  *src/nodeinfo/*: NodeInfo protocol implementation
+    -  *src/otel/*: OpenTelemetry integration utilities
     -  *src/runtime/*: Runtime utilities
     -  *src/shim/*: Platform abstraction layer
     -  *src/sig/*: Signature implementation

CHANGES.md

@@ -46,9 +46,30 @@ To be released.
      -  Added `KvStoreListEntry` interface.
      -  Implemented in `MemoryKvStore`.
 
+ -  Added `FedifySpanExporter` class that persists ActivityPub activity traces
+    to a `KvStore` for distributed tracing support.  This enables aggregating
+    trace data across multiple nodes in a distributed deployment, making it
+    possible to build debug dashboards that show complete request flows across
+    web servers and background workers.  [[#497], [#502]]
+
+     -  Added `@fedify/fedify/otel` module.
+     -  Added `FedifySpanExporter` class implementing OpenTelemetry's
+        `SpanExporter` interface.
+     -  Added `TraceActivityRecord` interface for stored activity data,
+        including `actorId` and `signatureDetails` fields for debug dashboard
+        support.
+     -  Added `SignatureVerificationDetails` interface for detailed signature
+        verification information.
+     -  Added `TraceSummary` interface for trace listing.
+     -  Added `FedifySpanExporterOptions` interface.
+     -  Added `GetRecentTracesOptions` interface.
+     -  Added `ActivityDirection` type.
+
 [#323]: https://github.com/fedify-dev/fedify/issues/323
+[#497]: https://github.com/fedify-dev/fedify/issues/497
 [#498]: https://github.com/fedify-dev/fedify/issues/498
 [#500]: https://github.com/fedify-dev/fedify/pull/500
+[#502]: https://github.com/fedify-dev/fedify/pull/502
 
 ### @fedify/nestjs
 

docs/manual/log.md

@@ -210,6 +210,14 @@ The `["fedify", "nodeinfo", "client"]` category is used for logging messages
 related to the NodeInfo client.  When you are curious about the NodeInfo client,
 you can check the log messages in this category with the `"error"` level.
 
+### `["fedify", "otel", "exporter"]`
+
+*This category is available since Fedify 1.10.0.*
+
+The `["fedify", "otel", "exporter"]` category is used for logging messages
+related to the `FedifySpanExporter`.  When span export to the `KvStore` fails,
+you can check the log messages in this category with the `"error"` level.
+
 ### `["fedify", "runtime", "docloader"]`
 
 *This category is available since Fedify 0.8.0.*

docs/manual/opentelemetry.md

@@ -394,3 +394,196 @@ can use `debugExporter.getActivities()` to access the captured activities for
 your debug dashboard or other observability tools.
 
 [SpanExporter]: https://open-telemetry.github.io/opentelemetry-js/interfaces/_opentelemetry_sdk_trace_base.SpanExporter.html
+
+
+Distributed trace storage with `FedifySpanExporter`
+---------------------------------------------------
+
+*This API is available since Fedify 1.10.0.*
+
+The example `FedifyDebugExporter` shown above stores activities in memory,
+which works well for single-process applications.  However, Fedify applications
+often run in distributed environments where:
+
+ -  The web server handling HTTP requests runs on different nodes than
+    the background workers processing the message queue.
+ -  Multiple worker nodes may process queued messages in parallel.
+ -  The debug dashboard itself may run on yet another node.
+
+In such environments, an in-memory exporter cannot aggregate traces across
+nodes.  Each node would only see its own spans, making it impossible to view
+the complete picture of a distributed trace.
+
+Fedify provides [`FedifySpanExporter`] which persists trace data to a
+[`KvStore`](./kv.md), enabling distributed tracing across multiple nodes.
+All nodes can write to the same storage, and your debug dashboard can query
+this shared storage to display complete traces.
+
+### Setting up `FedifySpanExporter`
+
+To use `FedifySpanExporter`, import it from the `@fedify/fedify/otel` module
+and configure it with a [`KvStore`](./kv.md):
+
+::: code-group
+
+~~~~ typescript twoslash [Deno]
+import type { KvStore, MessageQueue } from "@fedify/fedify";
+// ---cut-before---
+import { createFederation } from "@fedify/fedify";
+import { RedisKvStore } from "@fedify/redis";
+import { FedifySpanExporter } from "@fedify/fedify/otel";
+import {
+  BasicTracerProvider,
+  SimpleSpanProcessor,
+} from "@opentelemetry/sdk-trace-base";
+import Redis from "ioredis";
+
+const redis = new Redis();
+const kv = new RedisKvStore(redis);
+
+// Create the exporter that writes to KvStore
+const fedifyExporter = new FedifySpanExporter(kv, {
+  ttl: Temporal.Duration.from({ hours: 1 }),
+});
+
+const tracerProvider = new BasicTracerProvider();
+tracerProvider.addSpanProcessor(new SimpleSpanProcessor(fedifyExporter));
+
+const federation = createFederation<void>({
+  kv,
+  tracerProvider,
+// ---cut-start---
+  queue: null as unknown as MessageQueue,
+// ---cut-end---
+  // Omitted for brevity; see the related section for details.
+});
+~~~~
+
+~~~~ typescript [Node.js]
+import { createFederation } from "@fedify/fedify";
+import { RedisKvStore } from "@fedify/redis";
+import { FedifySpanExporter } from "@fedify/fedify/otel";
+import { NodeTracerProvider, SimpleSpanProcessor } from "@opentelemetry/sdk-trace-node";
+import Redis from "ioredis";
+
+const redis = new Redis();
+const kv = new RedisKvStore(redis);
+
+// Create the exporter that writes to KvStore
+const fedifyExporter = new FedifySpanExporter(kv, {
+  ttl: Temporal.Duration.from({ hours: 1 }),
+});
+
+const tracerProvider = new NodeTracerProvider();
+tracerProvider.addSpanProcessor(new SimpleSpanProcessor(fedifyExporter));
+
+const federation = createFederation({
+  kv,
+  tracerProvider,
+  // Omitted for brevity; see the related section for details.
+});
+~~~~
+
+:::
+
+### Querying stored traces
+
+The `FedifySpanExporter` provides methods to query stored trace data:
+
+~~~~ typescript twoslash
+import { MemoryKvStore } from "@fedify/fedify";
+import { FedifySpanExporter } from "@fedify/fedify/otel";
+const kv = new MemoryKvStore();
+const fedifyExporter = new FedifySpanExporter(kv);
+const traceId = "";
+// ---cut-before---
+// Get all activities for a specific trace
+const activities = await fedifyExporter.getActivitiesByTraceId(traceId);
+
+// Get recent traces (with optional limit)
+const recentTraces = await fedifyExporter.getRecentTraces({ limit: 100 });
+~~~~
+
+> [!NOTE]
+> The `~FedifySpanExporter.getRecentTraces()` method requires a `KvStore`
+> implementation that supports the `list()` method.  When using a store
+> that only provides `cas()` without `list()` support, this method will
+> return an empty array.
+
+Each `TraceActivityRecord` contains:
+
+ -  `traceId`: The OpenTelemetry trace ID
+ -  `spanId`: The OpenTelemetry span ID
+ -  `parentSpanId`: The parent span ID (if any)
+ -  `direction`: `"inbound"` or `"outbound"`
+ -  `activityType`: The ActivityPub activity type (e.g., `"Create"`, `"Follow"`)
+ -  `activityId`: The activity's ID URL
+ -  `actorId`: The actor ID URL (sender of the activity)
+ -  `activityJson`: The complete activity JSON
+ -  `verified`: Whether the activity was verified (for inbound activities)
+ -  `signatureDetails`: Detailed signature verification information
+    (for inbound activities), containing:
+     -  `httpSignaturesVerified`: Whether HTTP Signatures were verified
+     -  `httpSignaturesKeyId` (optional): The key ID used for HTTP signature
+        verification, if available
+     -  `ldSignaturesVerified`: Whether Linked Data Signatures were verified
+ -  `timestamp`: ISO 8601 timestamp
+ -  `inboxUrl`: The target inbox URL (for outbound activities)
+
+### Configuration options
+
+The `FedifySpanExporter` constructor accepts the following options:
+
+`ttl`
+:   The time-to-live for stored trace data.  If not specified, data will be
+    stored indefinitely (or until manually deleted).  This is useful for
+    automatically cleaning up old trace data:
+
+    ~~~~ typescript twoslash
+    import { MemoryKvStore } from "@fedify/fedify";
+    import { FedifySpanExporter } from "@fedify/fedify/otel";
+    const kv = new MemoryKvStore();
+    // ---cut-before---
+    const exporter = new FedifySpanExporter(kv, {
+      ttl: Temporal.Duration.from({ hours: 24 }),
+    });
+    ~~~~
+
+`keyPrefix`
+:   The key prefix for storing trace data in the `KvStore`.  Defaults to
+    `["fedify", "traces"]`.  You can customize this to avoid conflicts with
+    other data in the same `KvStore`:
+
+    ~~~~ typescript twoslash
+    import { MemoryKvStore } from "@fedify/fedify";
+    import { FedifySpanExporter } from "@fedify/fedify/otel";
+    const kv = new MemoryKvStore();
+    // ---cut-before---
+    const exporter = new FedifySpanExporter(kv, {
+      keyPrefix: ["myapp", "otel", "traces"],
+    });
+    ~~~~
+
+### `KvStore` requirements
+
+The `FedifySpanExporter` requires a [`KvStore`](./kv.md) that supports either
+the `list()` method (preferred) or the `cas()` method:
+
+ -  When `list()` is available, the exporter stores each activity record under
+    its own unique key, enabling efficient prefix scans without concurrency
+    issues.
+ -  When only `cas()` is available, the exporter uses compare-and-swap
+    operations to append records to a list, which works but may experience
+    contention under high load.
+ -  If neither method is available, the constructor throws an error.
+
+The following `KvStore` implementations support the required operations:
+
+ -  `MemoryKvStore` (supports both `list()` and `cas()`)
+ -  `RedisKvStore` from *@fedify/redis* (supports both `list()` and `cas()`)
+ -  `PostgresKvStore` from *@fedify/postgres* (supports `list()`)
+ -  `SqliteKvStore` from *@fedify/sqlite* (supports `list()`)
+ -  `DenoKvStore` from *@fedify/denokv* (supports both `list()` and `cas()`)
+ -  `WorkersKvStore` from *@fedify/cfworkers* (supports `list()`)
+
+[`FedifySpanExporter`]: https://jsr.io/@fedify/fedify/doc/otel/~/FedifySpanExporter

packages/fedify/deno.json

@@ -12,6 +12,7 @@
     "./testing": "./src/testing/mod.ts",
     "./vocab": "./src/vocab/mod.ts",
     "./webfinger": "./src/webfinger/mod.ts",
+    "./otel": "./src/otel/mod.ts",
     "./x/cfworkers": "./src/x/cfworkers.ts",
     "./x/denokv": "./src/x/denokv.ts",
     "./x/fresh": "./src/x/fresh.ts",

packages/fedify/package.json

@@ -118,6 +118,16 @@
       "require": "./dist/webfinger/mod.cjs",
       "default": "./dist/webfinger/mod.js"
     },
+    "./otel": {
+      "types": {
+        "import": "./dist/otel/mod.d.ts",
+        "require": "./dist/otel/mod.d.cts",
+        "default": "./dist/otel/mod.d.ts"
+      },
+      "import": "./dist/otel/mod.js",
+      "require": "./dist/otel/mod.cjs",
+      "default": "./dist/otel/mod.js"
+    },
     "./x/cfworkers": {
       "types": {
         "import": "./dist/x/cfworkers.d.ts",