@@ -4,6 +4,74 @@ These are engineering contributions and counterexamples, not product adoption
44claims or endorsements. Each case links to its public source and states what
55the evidence does not establish.
66
7+ ## Ecosystem audit: 33 verified usage-accounting findings (2026-09-06/07)
8+
9+ Over two days we deep-read 71 repositories across the Claude/Codex usage-tool
10+ long tail, LLM observability platforms, eval frameworks, gateways, and
11+ upstream pricing-data sources, and filed 33 evidence-backed findings — every
12+ one pinned to a commit SHA, verified by a runnable reproduction or a quoted
13+ static argument, and checked against the target's issue history for
14+ duplicates. The method is the product demo: the same hazard taxonomy and
15+ synthetic-fixture discipline that drive the AgentMeasure conformance pack
16+ were applied to other people's code.
17+
18+ What this establishes: these specific defects exist at these commits, and the
19+ audit method finds real, quantified accounting errors that the projects' own
20+ test suites missed. What it does not establish: that any of these projects
21+ endorses or uses AgentMeasure, that the maintainers will accept the fixes, or
22+ that every tool in the space has these defects (the majority of audited
23+ repos — including promptfoo, Portkey's model data, and tokscale — passed
24+ clean).
25+
26+ The hazard classes (with the number of confirmed instances found):
27+
28+ - ** Per-block re-summation** — Claude Code writes one JSONL line per content
29+ block, each carrying the same ` message.id ` and the same ` message.usage ` ;
30+ summing per line multiplies usage by block count. 9 confirmed instances,
31+ 2×–3× inflation (e.g. [ tokendash #37 ] ( https://github.com/zhangferry/tokendash/pull/37 ) ,
32+ [ agent-bill #1 ] ( https://github.com/yange0793-dot/agent-bill/pull/1 ) ,
33+ [ ccem #12 ] ( https://github.com/Genuifx/ccem/issues/12 ) ,
34+ [ samewrite #1 ] ( https://github.com/ipeterpetrus/samewrite/issues/1 ) ).
35+ - ** Re-emitted event re-summation** — Codex re-emits ` token_count ` events with
36+ identical totals after compaction/settings/rate-limit refreshes; naive
37+ per-event accumulation double-counts them. Verified on real corpora and
38+ fixed in 4 tools (e.g. [ tokenscope #6 ] ( https://github.com/stealthsrc/tokenscope/pull/6 ) ,
39+ [ token-vision #1 ] ( https://github.com/extrei/token-vision/issues/1 ) ,
40+ [ coding-agent-usage-tracker #1 ] ( https://github.com/avihut/coding-agent-usage-tracker/pull/1 ) ).
41+ - ** Cache pricing semantics** — Anthropic ` input_tokens ` excludes cache while
42+ OpenAI's includes it; conflating the two clamps input to zero or double-
43+ charges cache. 1h-TTL writes are 2× input, not 1.25×
44+ ([ claude-hud #758 ] ( https://github.com/jarrodwatts/claude-hud/pull/758 ) ,
45+ [ tokenguard analysis] ( https://github.com/QQSHI13/tokenguard ) — repo
46+ archived before filing, [ one-api-pro #13 ] ( https://github.com/modelbus/one-api-pro/issues/13 ) ).
47+ - ** Pricing-table drift** — embedded price tables diverging from published
48+ rates, 1.2×–3× per model ([ switchXprovider #1 ] ( https://github.com/shaheer-00/switchXprovider/pull/1 ) ,
49+ [ Tokdash #75 ] ( https://github.com/JingbiaoMei/Tokdash/pull/75 ) ,
50+ [ iris-eval #478 ] ( https://github.com/iris-eval/mcp-server/pull/478 ) ,
51+ [ litellm #40360 ] ( https://github.com/BerriAI/litellm/issues/40360 ) — in
52+ the industry's pricing source of record).
53+ - ** Loss on resume/fork** — sessions re-opened or forked lose or duplicate
54+ their history ([ ai-usage-inspector #1 ] ( https://github.com/Kud0o/ai-usage-inspector/issues/1 ) ,
55+ [ swarm #145 ] ( https://github.com/ra3orblade/swarm/issues/145 ) ).
56+
57+ ** Accepted so far: 1.** The swarm maintainer merged the fix in
58+ [ swarm PR #146 ] ( https://github.com/ra3orblade/swarm/pull/146 ) , crediting the
59+ report: * "reported and diagnosed by @roy-tong , who also supplied the patch
60+ this follows."* Everything else is open and unreviewed at the time of
61+ writing; several maintainers approved CI runs on our PRs
62+ ([ claude-hud] ( https://github.com/jarrodwatts/claude-hud/pull/757 ) ,
63+ [ Claude-Code-Agent-Monitor
64+ #328 ] ( https://github.com/hoangsonww/Claude-Code-Agent-Monitor/pull/328 ) ).
65+ Reproductions use fully synthetic fixtures derived from this project's
66+ reference corpus — no private logs were shared anywhere.
67+
68+ The full audit log, per-finding evidence, and the honest no-gap list (most
69+ audited projects passed) are maintained internally; the public trail is the
70+ linked PRs and issues above. If you want the fixture shapes that expose these
71+ classes in your own tool, they are in the
72+ [ conformance pack] ( ../conformance/pack/README.md ) — no AgentMeasure runtime
73+ required.
74+
775## Exposing reasoning tokens without adding them twice
876
977[ OpenLIT PR #1476 ] ( https://github.com/openlit/openlit/pull/1476 ) was merged on
0 commit comments