Scanner accounting wrong for current per-content-block transcript format (tools, prompts tab, turns)

[sdkkds]

Summary

The scanner's snapshot dedup assumes Claude Code writes growing-snapshot assistant records (several JSONL lines per response, the last a superset of the earlier ones). Current Claude Code writes one record per content block instead: a single response of [thinking] [text] [tool_use A] [tool_use B] is four disjoint records that share one message.id, each carrying the same final usage.

I measured 0 of 928 multi-record message.id groups as growing snapshots across real transcripts — all were disjoint. Token totals survive (every sibling carries the final usage), but three secondary metrics are silently wrong.

The three bugs

1. Top Tools / Skills undercount — scanner.py _evict_prior_snapshots

The dedup keeps only the last record of a (session_id, message_id) group and deletes the siblings' tool_calls. With one tool_use per record, every parallel tool call except the last is destroyed. Across real data, ~23% of multi-record groups carried tool_use blocks in evicted siblings.

Suggested fix: keep collapsing to one message row (summing siblings would multi-count tokens), but stop deleting the evicted siblings' tool_calls rows. The breakdown queries group by tool_name and never join messages, so the orphaned message_uuid is harmless, and full rescans stay idempotent because scan_file already clears a record's tool_calls by its own uuid before reinserting.

2. Prompts tab loses ~90% of prompts — db.py expensive_prompts

The join a.parent_uuid = u.uuid AND a.type='assistant' expects the first assistant record (whose parent is the user prompt) to survive — but eviction keeps the last record, whose parent_uuid points at a sibling. Measured join survival: ~13%.

Suggested fix: attribute each prompt to the first assistant record after it by timestamp within the session, rather than by parent_uuid — independent of which sibling survives. This raised join survival to ~95% on real data.

3. "Turns" inflated ~7x — db.py (overview_totals, project_summary, recent_sessions)

SUM(CASE WHEN type='user' THEN 1 ...) counts tool results, which Claude Code writes back as type='user' records. On real data this reported 12,638 turns where only 1,769 real prompts existed.

Suggested fix: add AND prompt_text IS NOT NULL to the three turn counts — the same filter expensive_prompts already uses.

Note on the test suite

tests/test_scanner_dedup.py fabricates the growing-snapshot format, so all three bugs pass CI today. A fixture captured from a current-format session (disjoint sibling records, parallel tool_use, tool_result-as-user) makes them visible.

Patch available

I have a tested implementation of all three fixes (new real-format fixture + the three changes; 75 tests passing, idempotency verified by triple full-rescan on real transcripts). Happy to open a PR if you'd like — let me know and I'll set up a fork, or I can paste the diff here.

[muckybuzzwoo]

Thanks for the unusually precise write-up — I reproduced all three on my fork's data (turns inflated ~7.9x; ~28% of tool_use blocks in non-final sibling records; 957/996 multi-record groups disjoint, matching your "0 of 928 growing snapshots"). #2 was already addressed in this fork via session+time-window prompt linkage. I implemented #1 and #3 based on your analysis in muckybuzzwoo/token-dashboard (release https://github.com/muckybuzzwoo/token-dashboard/releases/tag/v0.3.1) — fork-specific adaptation since the scanner batches dedup and uses materialised summary tables, so I re-point evicted siblings' tool_calls onto the surviving keeper (deduped by tool_use_id) rather than orphaning them, because several queries here join tool_calls to messages. Verified idempotent + token-neutral on real transcripts. Really appreciate the diagnosis — credited you in the fork notes. 🙏