Skip to content

Add OpenTelemetry span events and missing instrumentation #483

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Add OpenTelemetry span events and missing instrumentation #483

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
e16f22b
OpenTelemetry span events and missing instrumentation
dahlia Nov 22, 2025
8f47dd8
Fix example code based on review feedback
dahlia Nov 22, 2025
4547e9d
Rename span 'docloader.fetch' to 'activitypub.fetch_document' for con…
dahlia Nov 22, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions CHANGES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,36 @@ Version 1.10.0

To be released.

### @fedify/fedify

- Enhanced OpenTelemetry instrumentation with span events for capturing
detailed activity data. Span events now record complete activity JSON
payloads and verification status, enabling richer observability and
debugging capabilities without relying solely on span attributes
(which only support primitive values). [[#323]]

- Added `activitypub.activity.received` span event to the
`activitypub.inbox` span, recording the full activity JSON,
verification status (activity verified, HTTP signatures verified,
Linked Data signatures verified), and actor information.
- Added `activitypub.activity.sent` span event to the
`activitypub.send_activity` span, recording the full activity JSON
and target inbox URL.
- Added `activitypub.object.fetched` span event to the
`activitypub.lookup_object` span, recording the fetched object's
type and complete JSON-LD representation.

- Added OpenTelemetry spans for previously uninstrumented operations:
[[#323]]

- Added `activitypub.fetch_document` span for document loader operations,
tracking URL fetching, HTTP redirects, and final document URLs.
- Added `activitypub.verify_key_ownership` span for cryptographic
key ownership verification, recording actor ID, key ID, verification
result, and the verification method used.

[#323]: https://github.com/fedify-dev/fedify/issues/323


Version 1.9.1
-------------
Expand Down
36 changes: 33 additions & 3 deletions deno.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

156 changes: 156 additions & 0 deletions docs/manual/opentelemetry.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -174,7 +174,9 @@ spans:
| `activitypub.outbox` | Consumer | Dequeues the ActivityPub activity to send. |
| `activitypub.outbox` | Producer | Enqueues the ActivityPub activity to send. |
| `activitypub.parse_object` | Internal | Parses the Activity Streams object. |
| `activitypub.fetch_document` | Client | Fetches a remote JSON-LD document. |
| `activitypub.send_activity` | Client | Sends the ActivityPub activity. |
| `activitypub.verify_key_ownership` | Internal | Verifies actor ownership of a key. |
| `http_signatures.sign` | Internal | Signs the HTTP request. |
| `http_signatures.verify` | Internal | Verifies the HTTP request signature. |
| `ld_signatures.sign` | Internal | Makes the Linked Data signature. |
Expand All @@ -189,6 +191,47 @@ More operations will be instrumented in the future releases.
[Span kind]: https://opentelemetry.io/docs/specs/otel/trace/api/#spankind


Span events
-----------

In addition to spans, Fedify also records [span events] to capture rich,
structured data about key operations. Span events allow recording complex data
that wouldn't fit in span attributes (which are limited to primitive values).

The following span events are recorded:

| Event name | Recorded on span | Description |
|------------------------------------|------------------------------|----------------------------------------------------------------------------------|
| `activitypub.activity.received` | `activitypub.inbox` | Records full activity JSON and verification status when an activity is received. |
| `activitypub.activity.sent` | `activitypub.send_activity` | Records full activity JSON and delivery details when an activity is sent. |
| `activitypub.object.fetched` | `activitypub.lookup_object` | Records full object JSON when successfully fetched. |

### Event attributes

Each span event includes attributes with detailed information:

**`activitypub.activity.received` event attributes:**

- `activitypub.activity.json`: The complete activity JSON
- `activitypub.activity.verified`: Whether the activity was verified (`true`/`false`)
- `ld_signatures.verified`: Whether Linked Data Signatures were verified (`true`/`false`)
- `http_signatures.verified`: Whether HTTP Signatures were verified (`true`/`false`)
- `http_signatures.key_id`: The key ID used for HTTP signature verification

**`activitypub.activity.sent` event attributes:**

- `activitypub.activity.json`: The complete activity JSON being sent
- `activitypub.inbox.url`: The inbox URL where the activity was delivered
- `activitypub.activity.id`: The activity ID

**`activitypub.object.fetched` event attributes:**

- `activitypub.object.type`: The type URI of the fetched object
- `activitypub.object.json`: The complete object JSON

[span events]: https://opentelemetry.io/docs/concepts/signals/traces/#span-events


Semantic [attributes] for ActivityPub
-------------------------------------

Expand All @@ -209,6 +252,9 @@ for ActivityPub:
| `activitypub.actor.id` | string | The URI of the actor object. | `"https://example.com/actor/1"` |
| `activitypub.actor.key.cached` | boolean | Whether the actor's public keys are cached. | `true` |
| `activitypub.actor.type` | string[] | The qualified URI(s) of the actor type(s). | `["https://www.w3.org/ns/activitystreams#Person"]` |
| `activitypub.key.id` | string | The URI of the cryptographic key being verified. | `"https://example.com/actor/1#main-key"` |
| `activitypub.key_ownership.method` | string | The method used to verify key ownership (`owner_id` or `actor_fetch`). | `"actor_fetch"` |
| `activitypub.key_ownership.verified` | boolean | Whether the key ownership was successfully verified. | `true` |
| `activitypub.collection.id` | string | The URI of the collection object. | `"https://example.com/collection/1"` |
| `activitypub.collection.type` | string[] | The qualified URI(s) of the collection type(s). | `["https://www.w3.org/ns/activitystreams#OrderedCollection"]` |
| `activitypub.collection.total_items` | int | The total number of items in the collection. | `42` |
Expand All @@ -217,12 +263,16 @@ for ActivityPub:
| `activitypub.object.in_reply_to` | string[] | The URI(s) of the original object to which the object reply. | `["https://example.com/object/1"]` |
| `activitypub.inboxes` | int | The number of inboxes the activity is sent to. | `12` |
| `activitypub.shared_inbox` | boolean | Whether the activity is sent to the shared inbox. | `true` |
| `docloader.context_url` | string | The URL of the JSON-LD context document (if provided via Link header). | `"https://www.w3.org/ns/activitystreams"` |
| `docloader.document_url` | string | The final URL of the fetched document (after following redirects). | `"https://example.com/object/1"` |
| `fedify.actor.identifier` | string | The identifier of the actor. | `"1"` |
| `fedify.inbox.recipient` | string | The identifier of the inbox recipient. | `"1"` |
| `fedify.object.type` | string | The URI of the object type. | `"https://www.w3.org/ns/activitystreams#Note"` |
| `fedify.object.values.{parameter}` | string[] | The argument values of the object dispatcher. | `["1", "2"]` |
| `fedify.collection.cursor` | string | The cursor of the collection. | `"eyJpZCI6IjEiLCJ0eXBlIjoiT3JkZXJlZENvbGxlY3Rpb24ifQ=="` |
| `fedify.collection.items` | number | The number of items in the collection page. It can be less than the total items. | `10` |
| `http.redirect.url` | string | The redirect URL when a document fetch results in a redirect. | `"https://example.com/new-location"` |
| `http.response.status_code` | int | The HTTP response status code. | `200` |
| `http_signatures.signature` | string | The signature of the HTTP request in hexadecimal. | `"73a74c990beabe6e59cc68f9c6db7811b59cbb22fd12dcffb3565b651540efe9"` |
| `http_signatures.algorithm` | string | The algorithm of the HTTP request signature. | `"rsa-sha256"` |
| `http_signatures.key_id` | string | The public key ID of the HTTP request signature. | `"https://example.com/actor/1#main-key"` |
Expand All @@ -233,8 +283,114 @@ for ActivityPub:
| `object_integrity_proofs.cryptosuite` | string | The cryptographic suite of the object integrity proof. | `"eddsa-jcs-2022"` |
| `object_integrity_proofs.key_id` | string | The public key ID of the object integrity proof. | `"https://example.com/actor/1#main-key"` |
| `object_integrity_proofs.signature` | string | The integrity proof of the object in hexadecimal. | `"73a74c990beabe6e59cc68f9c6db7811b59cbb22fd12dcffb3565b651540efe9"` |
| `url.full` | string | The full URL being fetched by the document loader. | `"https://example.com/actor/1"` |
| `webfinger.resource` | string | The queried resource URI. | `"acct:[email protected]"` |
| `webfinger.resource.scheme` | string | The scheme of the queried resource URI. | `"acct"` |

[attributes]: https://opentelemetry.io/docs/specs/otel/common/#attribute
[OpenTelemetry Semantic Conventions]: https://opentelemetry.io/docs/specs/semconv/


Building observability tools with OpenTelemetry
------------------------------------------------

The OpenTelemetry instrumentation in Fedify provides a powerful foundation for
building custom observability tools. By implementing a custom [SpanExporter],
you can capture and process all the telemetry data generated by Fedify to build
tools like debug dashboards, activity monitors, or analytics systems.

### Example: ActivityPub debug dashboard

Here's an example of how you might implement a custom `SpanExporter` to capture
ActivityPub activities for a debug dashboard:

~~~~ typescript
import type { SpanExporter, ReadableSpan } from "@opentelemetry/sdk-trace-base";
import { ExportResultCode } from "@opentelemetry/core";

interface ActivityRecord {
direction: "inbound" | "outbound";
activity: unknown;
timestamp: Date;
verified?: boolean;
}

export class FedifyDebugExporter implements SpanExporter {
private activities: ActivityRecord[] = [];

export(spans: ReadableSpan[], resultCallback: (result: { code: ExportResultCode }) => void): void {
for (const span of spans) {
// Capture inbound activities
if (span.name === "activitypub.inbox") {
const event = span.events.find(
(e) => e.name === "activitypub.activity.received"
);
if (event && event.attributes) {
this.activities.push({
direction: "inbound",
activity: JSON.parse(
event.attributes["activitypub.activity.json"] as string
),
timestamp: new Date(span.startTime[0] * 1000),
verified: event.attributes["activitypub.activity.verified"] as boolean,
});
}
}

// Capture outbound activities
if (span.name === "activitypub.send_activity") {
const event = span.events.find(
(e) => e.name === "activitypub.activity.sent"
);
if (event && event.attributes) {
this.activities.push({
direction: "outbound",
activity: JSON.parse(
event.attributes["activitypub.activity.json"] as string
),
timestamp: new Date(span.startTime[0] * 1000),
});
}
}
}
resultCallback({ code: ExportResultCode.SUCCESS });
}

async forceFlush(): Promise<void> {
// Flush any pending data
}

async shutdown(): Promise<void> {
// Clean up resources
}

getActivities(): ActivityRecord[] {
return this.activities;
}
}
~~~~

### Integrating the custom exporter

To use the custom exporter, add it to your OpenTelemetry SDK configuration:

~~~~ typescript
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
import { SimpleSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { createFederation } from "@fedify/fedify";

const debugExporter = new FedifyDebugExporter();
const tracerProvider = new NodeTracerProvider();
tracerProvider.addSpanProcessor(new SimpleSpanProcessor(debugExporter));

const federation = createFederation({
kv: /* your KV store */,
tracerProvider,
});
~~~~

Now the `debugExporter` will receive all telemetry data from Fedify, and you
can use `debugExporter.getActivities()` to access the captured activities for
your debug dashboard or other observability tools.

[SpanExporter]: https://open-telemetry.github.io/opentelemetry-js/interfaces/_opentelemetry_sdk_trace_base.SpanExporter.html
2 changes: 2 additions & 0 deletions packages/fedify/deno.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@
"@cfworker/json-schema": "npm:@cfworker/json-schema@^4.1.1",
"@multiformats/base-x": "npm:@multiformats/base-x@^4.0.1",
"@opentelemetry/api": "npm:@opentelemetry/api@^1.9.0",
"@opentelemetry/core": "npm:@opentelemetry/core@^1.30.1",
"@opentelemetry/sdk-trace-base": "npm:@opentelemetry/sdk-trace-base@^1.30.1",
"@opentelemetry/semantic-conventions": "npm:@opentelemetry/semantic-conventions@^1.27.0",
"@phensley/language-tag": "npm:@phensley/language-tag@^1.9.0",
"@std/assert": "jsr:@std/assert@^0.226.0",
Expand Down
2 changes: 2 additions & 0 deletions packages/fedify/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,8 @@
"@logtape/logtape": "catalog:",
"@multiformats/base-x": "^4.0.1",
"@opentelemetry/api": "^1.9.0",
"@opentelemetry/core": "^1.30.1",
"@opentelemetry/sdk-trace-base": "^1.30.1",
"@opentelemetry/semantic-conventions": "^1.27.0",
"@phensley/language-tag": "^1.9.0",
"asn1js": "^3.0.5",
Expand Down
Loading
Loading