Skip to content

docs(casebook): ecosystem audit edition — 33 verified findings across 71 repos - #15

Merged
roy-tong merged 1 commit into
mainfrom
docs/ecosystem-audit-casebook
Sep 9, 2026
Merged

docs(casebook): ecosystem audit edition — 33 verified findings across 71 repos#15
roy-tong merged 1 commit into
mainfrom
docs/ecosystem-audit-casebook

Conversation

@roy-tong

@roy-tong roy-tong commented Sep 9, 2026

Copy link
Copy Markdown
Owner

What

Adds the ecosystem-audit edition to the measurement casebook: over 2026-09-06/07 we deep-read 71 repositories (usage tools, observability platforms, eval frameworks, gateways, pricing sources) and filed 33 evidence-backed usage-accounting findings. The new section summarizes the five hazard classes with confirmed-instance counts and links every claim to the public PR/issue.

Honesty rules kept

  • States explicitly what the audit does not establish (no adoption, no endorsement, most audited repos passed clean — promptfoo, Portkey model data, tokscale are named as passing).
  • One accepted artifact to date (swarm #145 → PR #146, maintainer-credited); all other items listed as open/unreviewed.
  • No private logs; all reproductions are synthetic fixtures from the conformance pack.

Why this belongs in the repo

The casebook is the product demo: the same taxonomy and fixture discipline that drive the conformance pack found quantified, verified defects that the targets' own test suites missed. Ends with a pointer to the fixture shapes for tool authors — no AgentMeasure runtime required.

@roy-tong
roy-tong merged commit 991e4e0 into main Sep 9, 2026
8 checks passed
@roy-tong
roy-tong deleted the docs/ecosystem-audit-casebook branch September 9, 2026 05:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant