Expand context attribution: MCP, /context fetch, per-span snapshots

[collincunn]

Summary

• Aligns trace-level token attribution with what claude -p "/context" reports — within ±1% for every category we can directly measure.
• Adds actual /context capture per trace via a detached background subprocess. ~$0 cost (local), ~1-3s async, zero session pollution.
• Adds per-span context_snapshot so cost dashboards can attribute billed tokens to categories with a single-row query (no JOIN to the trace).
• Closes the MCP gap from -99% to within ~15% of /context's MCP-tools-deferred bucket by harvesting mcp_instructions_delta and adding a per-tool overhead estimate.
• Distinguishes always-on vs deferred in every total_tokens field so naive sums no longer over-count tokens that don't actually bill (system_tools_deferred wasn't shipping in the request envelope when ToolSearch is enabled).

Headline numbers vs /context for the comet-ml/opik repo:

Category	/context	Ours	Delta
System prompt	8.0k	8.0k	exact (version-keyed) + exact (context_runtime)
System tools	17.6k	17.6k	exact (version-keyed) + exact (context_runtime)
System tools (deferred)	19.2k	19.2k	exact (version-keyed); marked informational
Custom agents	3.3k	3360	+1.8%
Memory files	2.9k	2806	-3.2%
Skills (menu)	2.1k	2112	+0.6%
MCP tools (deferred)	2.5k	2169	-13% (130/tool estimate)

What ships

cc.context_runtime (trace-level, async) — actual /context output captured per trace via claude --resume <id> --fork-session --no-session-persistence -p /context. Recursion-guarded with OPIK_CC_SKIP, 15s timeout, fails soft.

span.cc.context_snapshot (per-LLM-span, sync) — denormalized category breakdown so per-category cost is a single-row calc:

category_cost ≈ span.usage.input_tokens
              × span.cc.context_snapshot.categories[X]
              ÷ span.cc.context_snapshot.total_tokens
              × rate(cache_bucket)

Includes a cumulative messages category covering every prior user prompt + assistant output + tool_result in the session, so per-span total_tokens stays within ~1% of API billing even on 50+ turn sessions where conversation history accumulates.

cc.cc_builtin (trace-level, sync) — version-keyed estimates for system_prompt / system_tools / system_tools_deferred. Currently seeded for 2.1.150. Patch-version fallback in the same major.minor.

Slash-command skill loads — /opik:opik style invocations no longer fall through silent. Detected via <command-name>/<X></command-name> preamble + Base directory for this skill: follow-up text.

Plugin enablement filter — agents from disabled plugins (enabledPlugins[key] != true across managed/user/local/project settings) are filtered out of cc.agents. Was producing ghost rows.

Plugin catalog adoption — marketplace skills get catalog_body_tokens from ~/.claude/plugins/plugin-catalog-cache.json (CC's own tokenizer output) for cross-checking the estimate.

Calibration retuning — new content types: memory_file (2.4 chars/token), agent_frontmatter (3.1); skill_listing_menu retuned 3.9 → 3.0.

Always-on vs deferred semantics — every total_tokens now matches API billing; deferred_tokens surfaced separately. Eliminates the trap where naive SUM(total_tokens) over-counted by 19k+ per session.

Data flow: where each field is produced

The trace-level and per-span context fields are computed by separate code paths with different timing. Spans never receive /context data directly — that lands on the trace only. Read this if a dashboard is mixing the two.

┌─────────────────────────────────────────────────────────────────────────┐
│ User prompt → Claude responds                                            │
└─────────────────────────────────────────────────────────────────────────┘
                            │
                            ▼
┌─────────────────────────────────────────────────────────────────────────┐
│ flush() — runs every 5s + on Stop                                        │
│                                                                          │
│  ├─ processTranscriptEntries → builds Span structs                       │
│  ├─ buildContextSnapshot(state) → reads transcript + extractors:         │
│  │     extractMemorySnapshot, extractAgentsSnapshot,                     │
│  │     BuildSkillsSnapshot, extractToolsSnapshot,                        │
│  │     extractCCBuiltinSnapshot, cumulativeMessagesTokens               │
│  │                                                                       │
│  └─ for each span where usage != nil:                                    │
│       span.metadata.cc.context_snapshot = <snapshot>                     │
│       └─ source: "estimated_sync"                                        │
│                                                                          │
│  POST /spans/batch  ── spans go to Opik WITH context_snapshot embedded  │
└─────────────────────────────────────────────────────────────────────────┘
                            │
                            ▼ (Stop only)
┌─────────────────────────────────────────────────────────────────────────┐
│ onStop()                                                                 │
│  ├─ postTraceMetrics → PATCHes trace.metadata.cc.{memory, agents, …}    │
│  ├─ final PATCH trace (end_time, output)                                 │
│  └─ spawnDetachedContextFetch ── fire & forget; hook returns ~immediately│
└─────────────────────────────────────────────────────────────────────────┘
                            │
              ░░░ async, 1-3s later, detached subprocess ░░░
                            │
┌─────────────────────────────────────────────────────────────────────────┐
│ runContextFetchMode (the detached opik-logger child)                     │
│  ├─ exec claude --resume <id> --fork-session -p /context …               │
│  ├─ parseContextMarkdown → {categories, agents[], skills[], memory[], …} │
│  └─ PATCH trace.metadata.cc.context_runtime  ── TRACE only, never spans │
│       └─ source: "claude_context_command"                                │
└─────────────────────────────────────────────────────────────────────────┘

What each field actually contains

Field	Where	When	Source	Accuracy vs /context
span.metadata.cc.context_snapshot.categories.*	every LLM span	sync, at flush	our extractors + version constants	~1-15% drift
trace.metadata.cc.context_runtime.categories.*	trace	async, ~1-3s after Stop	parsed from /context output	exact
trace.metadata.cc.cc_builtin.summary.*	trace	sync, at Stop	version constants	exact for known versions
trace.metadata.cc.{memory,agents,skills,tools}.*	trace	sync, at Stop	our extractors	~1-15% drift

Why no back-propagation to spans

Considered: have the detached subprocess also walk every LLM span in the trace and PATCH each with context_runtime's exact numbers. Rejected — it would require one extra PATCH /spans/<id> per LLM call (5+ per multi-block turn), adding complexity for marginal accuracy gain (~1-15%). The dual-field setup with explicit source markers lets consumers pick:

• Per-span queries (cost attribution): use span.context_snapshot (no JOIN, sync estimate)
• Trace-level totals (what was actually in context): use trace.context_runtime (exact)

Honest limitations

These are called out so consumers know where the data is exact vs estimated.

1. Per-CC-version constants are manually maintained

cc.cc_builtin and the system-prompt/tools row of span.context_snapshot rely on a hardcoded table keyed by Claude Code CLI version (currently 2.1.150 only). The transcript doesn't carry these values — Claude Code holds the schemas internally and serializes them straight into the request envelope.

When CC ships a new major.minor with a different bundled prompt or tool catalog, the constants drift. Patch-version fallback (2.1.151 → use 2.1.150 row) handles minor changes; bigger bumps need a table update.

Mitigation: context_runtime (the async /context fetch) lands the exact numbers ~3s after Stop, regardless of CC version. Treat cc_builtin as a sync-time placeholder.

2. MCP per-tool schema overhead is a constant approximation

We can't read MCP tool schemas from the transcript — Claude Code calls tools/list at handshake, caches the schemas in RAM, and serializes them into the request envelope per turn. They never hit disk or the transcript.

Our cc.tools.by_source.mcp.estimated_schema_tokens uses a flat tool_count × 130 constant, calibrated against @modelcontextprotocol/server-everything (15 tools, /context = 2.5k tokens). Real per-tool schemas vary 80-430 tokens; the constant is ±15-30% per server.

Mitigation: marked estimated: true in the payload. context_runtime (async) has the exact MCP-tools-deferred number for the trace.

Considered alternative: spawning each MCP server at hook time to call tools/list ourselves. Rejected — too fragile (auth, OAuth flows, network) and too slow.

3. Sync context_snapshot drifts ~1-5% from the async context_runtime

The per-span snapshot uses synchronously-computed extractors at flush time, including a cumulative messages category that covers the conversation history. The trace-level context_runtime uses the actual /context output landed ~3s later via the detached subprocess.

The two are intentionally NOT reconciled — patching every LLM span retroactively when context_runtime lands would add complexity for marginal accuracy gain. They live side by side with explicit source markers:

• context_snapshot.source = "estimated_sync"
• context_runtime.source = "claude_context_command"

Pick which to use based on the query: per-span attribution → context_snapshot (no JOIN, ~1-5% drift); trace-level totals → context_runtime (exact).

Bucketing difference to watch: /context lumps loaded skill bodies into its Messages row. Our snapshot splits them into skills_loaded. Same tokens, different bucket names — don't sum skills_loaded and messages together (they're already disjoint by design).

4. Plugin catalog only covers @claude-plugins-official plugins

~/.claude/plugins/plugin-catalog-cache.json is populated by Claude Code for the official marketplace (~222 plugins). User-installed plugins from other marketplaces (e.g. opik@opik) aren't there.

Mitigation: catalog data is additive (catalog_body_tokens is a parallel field). Skills without catalog coverage still get the calibrated body_tokens estimate.

5. Per-turn /context fetch adds background CPU

Each Stop fires a detached claude subprocess that takes ~1-3s. Not on the user's path (the subprocess is detached via Setsid/CREATE_NEW_PROCESS_GROUP), but does add background work proportional to turn count.

For a 30-turn session: ~30 spawned subprocesses, ~60s of background CPU, ~200MB peak memory (one process at a time, with brief overlap if turns are <3s apart).

No API cost (/context is local) and no transcript pollution (--fork-session --no-session-persistence).

6. Naive SUM across traces in a session is wrong for most fields

Mixed additivity semantics — documented in detail [in the discussion thread but worth surfacing here]:

Field	Aggregation
cc.cc_builtin.*, cc.memory.*, cc.agents.*, cc.skills.summary.menu_tokens, cc.tools.*	MAX (session-stable)
cc.skills.summary.loaded_tokens, cc.context_runtime.*	LAST (cumulative snapshot)
cc.user_prompts.*, cc.assistant_text.*, cc.thinking.*, cc.tool_results.*	SUM (per-turn delta)
span.usage.*	SUM (per LLM call billed tokens)
cc.prior_assistant.*	LAST (already cumulative)

For billing analysis: use SUM(span.usage.*). For per-category cost: use span.context_snapshot + the formula above. The cc.* metadata describes structure/size, not what was billed N times.

Test plan

• go test ./src/... (58 tests pass)
• go vet ./src/... clean
• make build for darwin/arm64, darwin/amd64, linux/amd64, windows/amd64
• E2E: fresh session in ../opik, verify trace's cc.context_runtime.categories matches claude -p /context output
• E2E multi-turn with /opik:opik skill load — verify cc.skills.loaded populated, no user_prompts double-count
• E2E with --mcp-config everything — verify cc.tools.by_source.mcp.estimated_deferred_tokens lands within ~15% of /context's MCP row
• E2E: verify span.cc.context_snapshot.total_tokens matches span.usage.input_tokens + cache_read + cache_creation within ~1% (verified 3-turn skill-load session: 35,112 vs 34,957 = +0.4%)
• E2E: verify context_snapshot.categories.messages accumulates across turns and excludes loaded skill bodies (no double-count with skills_loaded)
• Detached subprocess: confirm parent transcript unchanged, no new session files created, hook returns in <100ms

🤖 Generated with Claude Code