feat(error-tracking): integrate Sentry for tool-call and transport failures

[alexkuzmik]

Details

Adds Sentry-based error reporting for the failures that need a human stack trace — tool-call bugs and transport crashes — alongside the existing analytics funnel. The two are independent: analytics is the low-cardinality install/usage funnel, Sentry is the stack-trace channel. Either can be opted out without affecting the other.

New module src/opik_mcp/error_tracking.py

• setup_sentry(settings) -> bool — initializes Sentry and binds the global scope. Pytest-guarded.
• capture_exception(exc, *, tags=, extras=, transaction=, fingerprint=) — pushes a fresh scope per event so concurrent HTTP-mode tool calls don't pollute each other's context.
• before_send filter: 30-event/process cap (kills retry-loop floods regardless of level — fatal included), pytest defence-in-depth.

DSN handling

• Hardcoded as a Settings.opik_mcp_sentry_dsn ClassVar — public ingest key, not a secret. Pydantic-settings skips ClassVars, so it's not env-overridable: prevents anyone redirecting crash reports to a foreign Sentry project. Pinned by test_settings_dsn_is_not_env_overridable.

What Sentry sees on a tool failure (read tool hitting a 500, for example)

• Tags: tool_name=read, error_kind=opik_http_5xx, entity_type=trace, id_kind=uuid, mcp_host=claude-code, mcp_client_version=…, plus globals (release, os_type, python_version, transport, installation_type matching opik SDK taxonomy: cloud/local/self-hosted, workspace, has_workspace_id, has_api_key, github_actions, source).
• Extras: duration_ms.
• Transaction: read (puts tool name in the issue listing next to the exception type).
• Fingerprint: ["{{ default }}", "read"] (splits issues per tool even when a shared helper raises).
• User: id = workspace UUID → workspace name → install_id fallback.

User-side failures are skipped at the capture site — not filtered server-side. The wrapper's _USER_SIDE_ERROR_KINDS allowlist excludes auth/permission/validation/not-found/missing-config/ollie-config/tool-args buckets; they're already counted in BI under explicit error_kind values, and Sentry only needs the failures that warrant a stack trace.

Transport-start crashes carry matching shape: phase=transport_start / error_kind=transport_crash / exception_type=<MRO-bucketed> / transport=stdio|streamable-http tags, startup transaction, ["{{ default }}", "startup", "transport_crash"] fingerprint.

Settings ValidationError stays analytics-only — the deploy pipeline hardcodes the values that drive validation, so this path is dev-only.

Smoke scripts in scripts/ for regression checks against the live Sentry project: sentry_smoke.py (canned exceptions through fake tool bodies — covers tags/transaction/fingerprint plumbing) and sentry_smoke_realistic.py (real OpikClient → httpx → socket failure against a closed local port — covers the real stack-trace shape).

Change checklist

• User facing
• Documentation updated (if needed)
• Tests added/updated (if needed)
• Breaking changes documented (if any)

Issues

• Resolves OPIK-6660

Testing

• uv run pytest -q — 553 passed, 2 skipped
• uv run ruff check . — clean
• uv run ruff format --check . — clean
• uv run mypy — clean (strict)
• Verified live with smoke scripts: events land in the opik-mcp Sentry project with the expected tags, transaction, fingerprint, and stack trace.

Documentation

README's ### Telemetry section is currently analytics-only; a follow-up PR will add a parallel ### Error tracking block documenting opt-out and what gets sent. Intentionally left out here to keep the diff focused on the implementation.

🤖 Generated with Claude Code