feat(scanner): ingest user-typed slash commands as Skill tool_calls - #12
Open
sebastientang wants to merge 6 commits into
Open
feat(scanner): ingest user-typed slash commands as Skill tool_calls#12sebastientang wants to merge 6 commits into
sebastientang wants to merge 6 commits into
Conversation
Derive project skill roots from distinct cwds in `messages` and feed them to `scan_catalog` alongside the three global roots. Innermost `.claude/skills` wins when cwds are nested, matching Claude Code's own resolution. Closes the first bullet of `docs/KNOWN_LIMITATIONS.md` and the first ideas-that-would-help item in `CONTRIBUTING.md`. Only skills invoked through `Task` with a skill-shaped `subagent_type` still show blank token counts. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Parses user-declared output-token budgets from SKILL.md body text
("Complete in <N output tokens" inline, or "## Token Budget" section)
and measures per-invocation output across tool_calls + messages using a
LEAD window: the attribution window for a Skill call ends at the next
Skill call in the same session, or session end. p50/p95 of that
distribution land as new columns on the Skills tab; the p50 cell goes
red when it exceeds the declared budget by more than 20%.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two measurement bugs surfaced on real data:
(1) auto-compaction sidechain agents fired inside the attribution
window and leaked tens of thousands of tokens into one-off skill
invocations.
(2) the "until next Skill or session end" boundary was far too wide
for skills that run once at session start, attributing the entire
downstream session to the skill.
Fix: exclude is_sidechain=1 from the JOIN, and cap the window at the
first real-user-typed main-chain message — prompt_chars > 0 with
prompt_text NOT matching any of seven system-injection prefixes
("Base directory for this skill:", "<system-reminder>",
"<command-name>", etc.) which Claude Code itself emits as user-role
messages.
Three new tests cover sidechain exclusion, real-user termination, and
meta-message non-termination.
Caveat surfaced in the help text: output_tokens includes tool_use JSON,
so a 2-5x gap over a text-only budget can be tool-call overhead, not
the skill misbehaving.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds skill_costs(db_path, pricing, since, until) which sums assistant- side cost (input + output + cache_read + cache_create) across each skill's attribution window, grouping tokens by (skill, model) so each bucket is priced with the matching model's rate. Falls back to tier- level pricing for unknown-but-named models and flags those rows with an asterisk. /api/skills now decorates each row with total_cost_usd + cost_estimated. The Skills tab sorts by total $ descending, turning the table from "overage ratios" into "which skills actually drive monthly spend." Surfaces the practical ranking that p50 alone can't: - skills with many invocations × modest p50 accumulate quietly - skills with huge p50 but one run can still be worth auditing - one column answers "where to look first" 3 new tests cover basic pricing, tier fallback, and multi-model windows. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Orchestrator skills dispatch subagents via Task/Agent. Until now their sidechain work was excluded from cost attribution, so the Skills tab under-reported their true cost by the full weight of whatever they dispatched. The new `skill_subagent_costs` joins each subagent's `agent_id` to the Skill window it started inside, exposes `subagent_cost_usd` / `subagent_output_tokens` / `total_with_subagents_usd` on `/api/skills`, and the Skills tab now ranks by total-with-subagents. Subagent window bounds diverge from the own-cost window on purpose: some interactive orchestrators prompt the user for input BEFORE dispatching, so closing at the first real-user message would miss the subagents entirely. Only the next Skill call in the session bounds the subagent window. Nested subagents fall out naturally — the inner agent's first sidechain message is itself inside the outer skill's window. Auto-compaction sidechains (`agent_id LIKE 'acompact%'`) are unconditionally excluded. Also: scanner.py's `_TARGET_FIELDS` only recognised the legacy "Task" tool name; Claude Code renamed it to "Agent" and historical rows had lost their `subagent_type` target. Adding "Agent" to the map fixes new scans; `python3 cli.py rescan-agent-targets` backfills the rest. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Slash-command invocations (`<command-name>/foo</command-name>` user records) previously never emitted a tool_calls row, so every user- invoked skill was under-counted in skill_breakdown / skill_costs / skill_actuals. Synthesize a tool_name='Skill' row at ingest time with target set to the command slug, keyed on the same uuid as the user message. Adds a `rescan-slash-commands` CLI subcommand for backfilling existing DBs without a full JSONL re-read. Hardens the attribution-window prefix filter against the command-message-first ordering observed in the wild: some sessions emit the `<command-message>` user record BEFORE the `<command-name>` record, and the previous filter dropped the first assistant turn in those cases. Rewritten to match either ordering. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
sebastientang
force-pushed
the
pr/slash-command-scanner
branch
from
April 26, 2026 11:03
499c9ab to
f9a49d9
Compare
muckybuzzwoo
added a commit
to muckybuzzwoo/token-dashboard
that referenced
this pull request
May 13, 2026
…mmand scanner Integrates nateherkai#12. Adds skill_budgets module (p50/p95/cost/subagent-attribution), slash-command tool_call synthesis, Agent target rename, project-local skills (supersedes #6, nateherkai#11). # Conflicts: # token_dashboard/scanner.py
muckybuzzwoo
added a commit
to muckybuzzwoo/token-dashboard
that referenced
this pull request
May 22, 2026
…e-invoked Adapts upstream PR nateherkai#19 (wolff-singleton) to this fork, with one behaviour-changing adjustment to avoid double-counting against the PR nateherkai#12 slash-command synthesiser already in this fork. Changes: - New messages.attribution_skill column, populated from each JSONL record's top-level attributionSkill field. Verified against real JSONLs: present on every assistant message inside an active slash- command session, plugin-namespaced (e.g. "buzzwoo-memory:memory- review"). Migration clears messages/tool_calls/files AND summary_meta (so the materialised summary tables get fully rebuilt instead of serving stale data) and forces a full rescan on the next launch. Settings, plan, and dismissed_tips tables are preserved. - skill_breakdown() now returns two counts per row: manual_sessions (distinct sessions where attribution_skill was set, i.e. the user typed a slash command) and tool_invocations (Skill tool_use blocks Claude emitted itself, typically Task/Agent subagents). Combined total drives ordering. - Skills route renamed "Skills & Commands". KPI tiles split into "You ran" / "Claude invoked". Table gains two columns matching. Fork's existing budget / p50 / p95 / total-$ / total-inc-subagents columns preserved (from PR nateherkai#12). Chart uses groupedBarChart for the two-series view, with a single-series fallback when only one category has data in the window. De-dupe fix vs. PR nateherkai#12: synthesised Skill rows from PR nateherkai#12 live on user-type messages; real tool_use rows live on assistant-type messages. The merged skill_breakdown's tool_inv CTE adds AND EXISTS (SELECT 1 FROM messages m WHERE m.uuid = tc.message_uuid AND m.type = 'assistant') so PR nateherkai#12's synthesised rows no longer double-count against PR nateherkai#19's attribution_skill counts. Verified on a real DB: 276 synthesised rows on user messages were correctly filtered out of "Claude invoked", while 51 real rows on assistant messages still count. PR nateherkai#12's rescan-slash-commands one-shot remains for back-filling DBs that pre-date this column. Test updates: SkillBreakdownTests fixture extended to cover both manual-only skills, tool-only skills, the COALESCE merge for skills with both, and an explicit de-dupe guard test that places a synthesised Skill row on a user-type message and asserts it does NOT inflate tool_invocations. The two server-side fixture files (test_server_skills_budget.py) had to parent their fake tool_calls rows to a real assistant message uuid so the new EXISTS join resolves. Upstream PR: nateherkai#19
This was referenced May 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Slash-command invocations (
<command-name>/foo</command-name>user records) previously never emitted atool_callsrow, so user-invoked skills were undercounted inskill_breakdown/skill_costs/skill_actuals.What this adds
<command-name>/foo</command-name>payload, it emits atool_name='Skill'row withtool_input='{"skill_name":"foo"}'keyed on the same uuid as the user message. From then on the skill appears in/api/skillsexactly like a programmatic Skill call.rescan-slash-commandsCLI subcommand — backfills existing DBs by re-scanning themessagestable only (no JSONL re-read needed).<command-message>user record arrives before the<command-name>record. The previous filter dropped the first assistant turn in those cases. Rewritten to match either ordering.Tests
15 new tests across:
tests/test_scanner_slash_commands.py(177 lines) — happy path, malformed tags, sidechain ignored, command-message-first orderingtests/test_scanner_parse.py(+74 lines) — pure regex coveragetests/test_skill_budgets.py(+114 lines) —SlashCommandAttributionTests(verifies the synthetic row plays correctly with the budget-attribution code from feat(skills): total $ column, budget-vs-actual flagging, subagent attribution #11)All 117 tests green:
python3 -m unittest discover tests.