Per-span Claude Code attribution metadata under cc.*

[collincunn]

Summary

Stand up a Go engine that scans Claude Code transcripts and emits
per-span structured token-attribution metadata to Opik. Every span on
every trace now carries the same domain snapshots under
`metadata.cc.{skills, tools, memory, thinking, tool_results,
user_prompts, file_attachments, prior_assistant, assistant_text,
llm_call, git, identity}` — so analytics can drill into "how much of
this LLM call's context was skills vs MCP catalog vs prior assistant
output" without re-parsing the transcript.

Design

• thread = session, trace = turn, span = message-content-block —
  reuses the existing shape from the prior PR.
• cc.llm_call tags every span with `{message_id, block_index,
  block_kind, measured_output_tokens, attributed_output_tokens}` so
  multi-block messages (thinking + text + tool_use sharing one
  `message.usage`) can be reconstructed without naive `GROUP BY
  span.name` looking like 100% thinking. Σattributed over a message_id
  always equals `anchor.usage.completion_tokens` exactly.
• Per-block attribution: text and tool_use blocks are directly
  measured (chars→tokens with calibrated per-type ratios); thinking
  blocks get the leftover. Edge cases (measured > total,
  no-thinking-blocks) are clamped/scaled to preserve the sum
  invariant.
• Calibrated tokenizer under `src/attribution.go::tokEstimateAs`
  with per-content-type chars/token ratios drawn from 643 real
  Claude Code transcript samples vs Anthropic count_tokens. Median
  error 19.3% (chars/4) → 9.9%. count_tokens is only called offline in
  `tools/calibration/`, not at runtime.
• Deterministic span IDs via `toV7(messageUUID)` so flushes are
  idempotent (no duplicate Skill-Loaded spans across flushes).
• Skill body resolution walks Skill tool_use → tool_result →
  next user-text to get the canonical body, hashes it (per-skill
  SHAs, not the listing-blob hash), and tracks loaded vs available
  separately. On-disk `resolveSkillBody` runs project → user → plugin
  paths.

Domain snapshot conventions

• One `summary` block per domain, with at least `total_tokens` (or
  `schema_tokens` for tools). Detail arrays separately.
• No timestamps anywhere under `cc.*` (deliberate — the user
  explicitly called these out as noise).
• Enums over booleans (`source: "listing"|"bundled"`).
• Per-event hook only when not already named on the span (e.g.
  `cc.skills.load` on a Skill tool_use span; `cc.tool.*` on an MCP
  tool_use span).
• Trace metadata mirrors the span metadata shape exactly under
  `metadata.cc.*` — same keys, same shape.

Critical fixes (this PR)

#	Fix	Before	After
1	MCP toggle honored	union-of-all-deltas, removals ignored	catalog replays in order; toggling shrinks `cc.tools.available_count`
2	Real `schema_tokens`	`tool_count × 15` everywhere	per-server from real `addedLines` text
3	Skill bodies not double-counted	146K skill body inflated `cc.user_prompts` to 119K	excluded via `skillBodyTextSet`
4	`cc.*` namespace cleanup	10+ top-level git + identity keys	grouped under `cc.git.` and `cc.identity.`
5	ToolSearch results counted	normal tool_result path missed the tool_reference shape; delta path not paired	both shapes counted via `pendingToolSearchID` pairing
6	`cc.thinking` matches attribution	recomputed per-entry, diverged by 2× on multi-block messages	shares `DeduplicateUsage` source so sums are guaranteed equal
7	MCP span display names	`mcp__server__tool`	bare tool name + `cc.tool.{name, server, full, source}`
8	MCP primitive tool_results no longer silently dropped	`toolUseResult` written as a string failed `json.Unmarshal`; `ReadTranscript` skipped the entry	custom `UnmarshalJSON` tolerates both string and object shapes

#8 is the worst kind — silent. Any MCP tool that returns a primitive
(jira_get_all_projects, list_branches, etc.) was completely invisible
to the engine across all traces before this PR.

Verification

Validated on two end-to-end sessions on the user's account:

• Pre-fix thread `b4c5e4c2…` exhibited every bug: 191 tools every
  trace despite the user toggling MCPs off, `schema_tokens = count × 15`
  uniform, `cc.user_prompts.total_tokens = 119,361` on the
  claude-api turn, `thinking ≠ attribution sums`, ToolSearch
  uncounted.
• Post-fix thread `356252c2…` (fresh session, new binary): 82 → 131 →
  45 `available_count` as the user toggled servers, real per-server
  schema_tokens, `cc.user_prompts.total_tokens = 7` on the same
  claude-api load, attribution math green on every LLM call.

`tools/verify.py` is a permanent smoke script:

```bash
SID= OPIK_KEY= python3 tools/verify.py
```

Runs B1 (required domains), C7 (tools sum), D3 (attribution invariant),
D4 (per-category non-degenerate), H1–H4 (no legacy keys / timestamps /
Skill Loaded spans). Returns non-zero on regression.

`src/dryrun_test.go` is an offline test harness — point it at a
captured transcript via `OPIK_DRY_TRANSCRIPT` and it dumps every
domain snapshot without touching Opik.

Test plan

• Start a fresh Claude Code session; confirm `cc.tools.summary.available_count`
  matches the actual catalog size (not 191).
• Toggle an MCP off mid-session; confirm `available_count` shrinks
  on the next trace.
• Load a large skill (`/claude-api`); confirm
  `cc.skills.loaded_tokens` carries the body and
  `cc.user_prompts.total_tokens` stays tiny.
• Invoke an MCP that returns a primitive (e.g.
  `mcp__GitHub__list_branches`); confirm the span appears with
  `cc.tool.server = "GitHub"` and the result tokens land in
  `cc.tool_results.by_tool`.
• Run `tools/verify.py` against the resulting trace — headline
  checks should all pass.

Out of scope / known gaps

• Real per-tool JSON schema cost (only available when Claude actually
  loads schemas via ToolSearch; we measure the addedLines/addedNames
  proxy).
• ToolSearch re-add deltas with empty addedNames/addedLines contribute
  0 tokens — re-enabling already-known tools doesn't create new model
  context cost, so this is probably correct, but worth a follow-up.

🤖 Generated with Claude Code