Add OpenTelemetry span events and missing instrumentation

[dahlia]

Summary

Enhanced OpenTelemetry observability by adding span events to capture rich activity data and instrumenting previously uncovered operations.

This PR implements the alternative approach proposed in #323, leveraging existing OpenTelemetry infrastructure instead of creating a new FederationObserver interface.

Changes

Span events added

Span events now record complete activity JSON payloads and verification status, enabling richer observability without relying solely on span attributes (which only support primitive values):

• activitypub.activity.received in the activitypub.inbox span

  • Records full activity JSON
  • Verification status (activity verified, HTTP signatures verified, LD signatures verified)
  • Actor information

• activitypub.activity.sent in the activitypub.send_activity span

  • Records activity JSON
  • Target inbox URL

• activitypub.object.fetched in the activitypub.lookup_object span

  • Records object type
  • Complete JSON-LD representation

New spans added

• docloader.fetch for document loader operations

  • Tracks URL fetching
  • HTTP redirects
  • Final document URLs

• activitypub.verify_key_ownership for key ownership verification

  • Records actor ID, key ID
  • Verification result and method used

Test coverage

Added comprehensive test coverage using a new TestSpanExporter helper in packages/fedify/src/testing/otel.ts to verify all instrumentation is properly recorded:

• Inbox handler span events
• Outbound activity sending span events
• Object lookup span events
• Document loader spans (including redirect handling)
• Key ownership verification spans

Documentation

Updated docs/manual/opentelemetry.md with:

• New spans and their attributes
• Span events section with detailed attribute documentation
• Example FedifyDebugExporter implementation showing how to build custom observability tools

Use cases

This enhancement enables developers to:

1. Build custom debugging dashboards by implementing custom SpanExporters that process span events
2. Inspect full activity payloads without modifying Fedify's core logging
3. Track verification failures with detailed context about what was verified and how
4. Monitor federation performance with detailed document fetching metrics

Breaking changes

None. This is a backward-compatible enhancement that adds new observability data points.

Resolves #323

[gemini-code-assist]

Summary of Changes

Hello @dahlia, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the observability of the Fedify project by integrating more comprehensive OpenTelemetry tracing. It introduces detailed span events to capture rich, structured data like full activity payloads and verification statuses, which was previously not possible with standard span attributes. Additionally, new spans have been added to cover critical operations such as document loading and cryptographic key ownership verification, providing deeper insights into the system's behavior and performance. These enhancements empower developers to build more sophisticated debugging and monitoring tools.

Highlights

• Enhanced OpenTelemetry Instrumentation with Span Events: Detailed activity JSON payloads and verification statuses are now captured as span events, providing richer observability than traditional span attributes which are limited to primitive values. This includes events for received activities, sent activities, and fetched objects.
• New OpenTelemetry Spans for Key Operations: Introduced new spans for previously uninstrumented operations: docloader.fetch to track URL fetching and redirects, and activitypub.verify_key_ownership to monitor cryptographic key ownership verification processes.
• Comprehensive Test Coverage for Observability: Added extensive test coverage for the new instrumentation using a custom TestSpanExporter, ensuring that all new spans and events are properly recorded and verifiable.
• Updated OpenTelemetry Documentation: The docs/manual/opentelemetry.md file has been updated to reflect the new spans and span events, including detailed attribute documentation and an example FedifyDebugExporter to guide developers in building custom observability tools.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request significantly enhances the observability of Fedify by introducing new OpenTelemetry spans and, more importantly, span events. The added instrumentation for document loading, key ownership verification, and various ActivityPub operations will provide much richer data for debugging and monitoring. The use of span events to capture full JSON payloads is a great approach to overcome the limitations of span attributes.

The changes are well-implemented, with comprehensive test coverage using the new TestSpanExporter helper, which is a nice addition for testing. The documentation has also been thoroughly updated to reflect these new features, including a helpful example of a custom SpanExporter.

I have a couple of minor suggestions to improve the code example in the documentation for clarity and correctness. Overall, this is an excellent contribution that greatly improves the project's observability capabilities.

[gemini-code-assist]

The verified property on ActivityRecord is mandatory. This leads to hardcoding it to true for outbound activities on line 352, which can be misleading as outbound activities aren't "verified" in the same way inbound ones are. The underlying activitypub.activity.sent event doesn't include a verified attribute.

Consider making the verified property optional. This would allow you to remove the hardcoded verified: true from the outbound activity record creation, more accurately reflecting the data available from the span event.

Suggested change

	interface ActivityRecord {
	direction: "inbound" | "outbound";
	activity: unknown;
	timestamp: Date;
	verified: boolean;
	}
	interface ActivityRecord {
	direction: "inbound" | "outbound";
	activity: unknown;
	timestamp: Date;
	verified?: boolean;
	}

[gemini-code-assist]

The type annotation for resultCallback is a bit loose. It can be made more specific to improve type safety and clarity for users copying this example. The result object has a code property of type ExportResultCode.

Suggested change

	export(spans: ReadableSpan[], resultCallback: (result) => void): void {
	export(spans: ReadableSpan[], resultCallback: (result: { code: ExportResultCode }) => void): void {