Skip to content

Commit 991e4e0

Browse files
roy-tongRoy tong
andauthored
docs(casebook): ecosystem audit edition — 33 verified findings, method as demo (#15)
Co-authored-by: Roy tong <tongxiarui@elliwooddeMacBook-Air.local>
1 parent 97a20ef commit 991e4e0

1 file changed

Lines changed: 68 additions & 0 deletions

File tree

campaigns/measurement-casebook.md

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,74 @@ These are engineering contributions and counterexamples, not product adoption
44
claims or endorsements. Each case links to its public source and states what
55
the evidence does not establish.
66

7+
## Ecosystem audit: 33 verified usage-accounting findings (2026-09-06/07)
8+
9+
Over two days we deep-read 71 repositories across the Claude/Codex usage-tool
10+
long tail, LLM observability platforms, eval frameworks, gateways, and
11+
upstream pricing-data sources, and filed 33 evidence-backed findings — every
12+
one pinned to a commit SHA, verified by a runnable reproduction or a quoted
13+
static argument, and checked against the target's issue history for
14+
duplicates. The method is the product demo: the same hazard taxonomy and
15+
synthetic-fixture discipline that drive the AgentMeasure conformance pack
16+
were applied to other people's code.
17+
18+
What this establishes: these specific defects exist at these commits, and the
19+
audit method finds real, quantified accounting errors that the projects' own
20+
test suites missed. What it does not establish: that any of these projects
21+
endorses or uses AgentMeasure, that the maintainers will accept the fixes, or
22+
that every tool in the space has these defects (the majority of audited
23+
repos — including promptfoo, Portkey's model data, and tokscale — passed
24+
clean).
25+
26+
The hazard classes (with the number of confirmed instances found):
27+
28+
- **Per-block re-summation** — Claude Code writes one JSONL line per content
29+
block, each carrying the same `message.id` and the same `message.usage`;
30+
summing per line multiplies usage by block count. 9 confirmed instances,
31+
2×–3× inflation (e.g. [tokendash #37](https://github.com/zhangferry/tokendash/pull/37),
32+
[agent-bill #1](https://github.com/yange0793-dot/agent-bill/pull/1),
33+
[ccem #12](https://github.com/Genuifx/ccem/issues/12),
34+
[samewrite #1](https://github.com/ipeterpetrus/samewrite/issues/1)).
35+
- **Re-emitted event re-summation** — Codex re-emits `token_count` events with
36+
identical totals after compaction/settings/rate-limit refreshes; naive
37+
per-event accumulation double-counts them. Verified on real corpora and
38+
fixed in 4 tools (e.g. [tokenscope #6](https://github.com/stealthsrc/tokenscope/pull/6),
39+
[token-vision #1](https://github.com/extrei/token-vision/issues/1),
40+
[coding-agent-usage-tracker #1](https://github.com/avihut/coding-agent-usage-tracker/pull/1)).
41+
- **Cache pricing semantics** — Anthropic `input_tokens` excludes cache while
42+
OpenAI's includes it; conflating the two clamps input to zero or double-
43+
charges cache. 1h-TTL writes are 2× input, not 1.25×
44+
([claude-hud #758](https://github.com/jarrodwatts/claude-hud/pull/758),
45+
[tokenguard analysis](https://github.com/QQSHI13/tokenguard) — repo
46+
archived before filing, [one-api-pro #13](https://github.com/modelbus/one-api-pro/issues/13)).
47+
- **Pricing-table drift** — embedded price tables diverging from published
48+
rates, 1.2×–3× per model ([switchXprovider #1](https://github.com/shaheer-00/switchXprovider/pull/1),
49+
[Tokdash #75](https://github.com/JingbiaoMei/Tokdash/pull/75),
50+
[iris-eval #478](https://github.com/iris-eval/mcp-server/pull/478),
51+
[litellm #40360](https://github.com/BerriAI/litellm/issues/40360) — in
52+
the industry's pricing source of record).
53+
- **Loss on resume/fork** — sessions re-opened or forked lose or duplicate
54+
their history ([ai-usage-inspector #1](https://github.com/Kud0o/ai-usage-inspector/issues/1),
55+
[swarm #145](https://github.com/ra3orblade/swarm/issues/145)).
56+
57+
**Accepted so far: 1.** The swarm maintainer merged the fix in
58+
[swarm PR #146](https://github.com/ra3orblade/swarm/pull/146), crediting the
59+
report: *"reported and diagnosed by @roy-tong, who also supplied the patch
60+
this follows."* Everything else is open and unreviewed at the time of
61+
writing; several maintainers approved CI runs on our PRs
62+
([claude-hud](https://github.com/jarrodwatts/claude-hud/pull/757),
63+
[Claude-Code-Agent-Monitor
64+
#328](https://github.com/hoangsonww/Claude-Code-Agent-Monitor/pull/328)).
65+
Reproductions use fully synthetic fixtures derived from this project's
66+
reference corpus — no private logs were shared anywhere.
67+
68+
The full audit log, per-finding evidence, and the honest no-gap list (most
69+
audited projects passed) are maintained internally; the public trail is the
70+
linked PRs and issues above. If you want the fixture shapes that expose these
71+
classes in your own tool, they are in the
72+
[conformance pack](../conformance/pack/README.md) — no AgentMeasure runtime
73+
required.
74+
775
## Exposing reasoning tokens without adding them twice
876

977
[OpenLIT PR #1476](https://github.com/openlit/openlit/pull/1476) was merged on

0 commit comments

Comments
 (0)