Skip to content

Commit 26576dd

Browse files
Eilodonclaude
andauthored
docs(readme): refresh stale measured numbers and undercounted lists (#49)
- Bump the "Proof, not promises" table and repo_overview example to the current 237 files / 3,583 symbols, hub/dead-code/edge-coverage/complexity percentages, and add the avg_distance (Martin/OOD) row that was missing. - Fitness check now measures 11 metrics, not 10 (avg_distance was added without updating the table). - Note that thresholds.toml already declares two [[boundaries]] rules, not just the one shown as an example. - Testing section said "Five CI jobs" — ci.yml actually runs eight (no-stack-graphs-formal, otel-http-features, and fitness-check were undocumented). - Further reading's benchmark list was missing four already-Implemented benchmarks (b3, b6, b7, b12). Co-authored-by: Eilodon <noreply@anthropic.com>
1 parent 4476f0a commit 26576dd

1 file changed

Lines changed: 13 additions & 9 deletions

File tree

README.md

Lines changed: 13 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -112,7 +112,7 @@ This repo ships ready-made config for Claude Code (`.mcp.json`), Cursor (`.curso
112112

113113
```
114114
agent: repo_overview()
115-
233 files, 3,488 symbols, indexing_phase=ready
115+
237 files, 3,583 symbols, indexing_phase=ready
116116
117117
agent: "I need to change getUserByEmail"
118118
→ locate("getUserByEmail") # find the file + symbol metadata
@@ -132,11 +132,12 @@ Every number below is measured by pointing CALM's own `fitness_report`/`repo_ove
132132

133133
| Metric | Measured value |
134134
|---|---|
135-
| Codebase indexed | **233 files, 3,488 symbols** — 15 languages present in this repo alone |
136-
| Hub concentration (`hub_pct`) | 7.5% — 169 hub symbols (gate: ≤ 20%) |
137-
| Dead-code rate (`dead_code_pct`, coverage-aware) | 5.0% (gate: ≤ 10%) |
138-
| Edge coverage (`edge_coverage_pct`) | 74.9% of symbols have at least one call edge (gate: ≥ 60%) |
139-
| High-complexity functions (`high_complexity_pct`) | 2.9% (gate: ≤ 15%) |
135+
| Codebase indexed | **237 files, 3,583 symbols** — 15 languages present in this repo alone |
136+
| Hub concentration (`hub_pct`) | 7.6% — 174 hub symbols (gate: ≤ 20%) |
137+
| Dead-code rate (`dead_code_pct`, coverage-aware) | 5.6% (gate: ≤ 10%) |
138+
| Edge coverage (`edge_coverage_pct`) | 70.1% of symbols have at least one call edge (gate: ≥ 60%) |
139+
| High-complexity functions (`high_complexity_pct`) | 2.8% (gate: ≤ 15%) |
140+
| Architecture fit (`avg_distance`, Martin/OOD) | 0.27 average distance from the main sequence (gate: ≤ 1.00) |
140141
| Ambiguous symbol boundaries (`boundary_ambiguous_count`) | 0 (gate: ≤ 0) |
141142
| Architecture boundary violations (`boundary_violations`) | 0 (gate: ≤ 0) — the `watcher → tools` import previously flagged here was fixed by relocating the shared `RwLockExt`/`LockExt` traits it needed out of `tools/common.rs` into their own `sync_ext` module |
142143
| Token efficiency vs. a naive read-the-files baseline | `source` **241x** · `edit_context` **193x** · `locate` **29x** · `callers` **1.0x** — median 111x across the four benchmark tasks ([methodology](benchmarks/b4_token_efficiency/)) |
@@ -226,7 +227,7 @@ Distinct from the `tools` above — MCP Prompts (`prompts/list`, `prompts/get`)
226227

227228
Run for real in `.github/workflows/ci.yml`'s `fitness-check` job on every push/PR — `calm index` first (a fresh checkout has no `.calm/index.db` yet), then `calm fitness-check --project-root . --config thresholds.toml`. That `--config` flag is not optional: without it, `[[boundaries]]` and `[config_drift]` are silently treated as "no rules declared" rather than erroring — only the numeric thresholds have a real default.
228229

229-
`calm fitness-check` measures 10 metrics against thresholds declared in `thresholds.toml`:
230+
`calm fitness-check` measures 11 metrics against thresholds declared in `thresholds.toml`:
230231

231232
| Metric | What it measures | Default threshold |
232233
|---|---|---|
@@ -237,6 +238,7 @@ Run for real in `.github/workflows/ci.yml`'s `fitness-check` job on every push/P
237238
| `hotspot_risk` | Highest hotspot score in the codebase | ≤ 0.75 |
238239
| `edge_coverage_pct` | % of symbols with at least one call edge | ≥ 60% |
239240
| `high_complexity_pct` | % of functions/methods with McCabe cyclomatic complexity > 10 (AST-based; Tier-0.5 languages always report complexity 1) | ≤ 15.0% |
241+
| `avg_distance` | Martin/OOD average distance from the main sequence — how far each file's abstractness sits from the ideal implied by its instability (Ca/Ce) | ≤ 1.00 |
240242
| `boundary_violations` | Count of `import_edges` violating a declared `[[boundaries]]` rule | ≤ 0 |
241243
| `boundary_ambiguous_count` | Count of symbols with an ambiguous line boundary (shared with a neighbor) — `edit_symbol` replace on these is refused until resolved | ≤ 0 |
242244
| `config_drift_count` | Count of doc file-path references (declared via `[config_drift].doc_paths`) pointing at nothing real | ≤ 0 |
@@ -256,6 +258,8 @@ reason = "indexer (extraction) must stay upstream of analysis (dead-code, hotspo
256258

257259
`calm fitness-check` reports each violation concretely (the real from/to path, the rule, and the reason) outside `--json` mode; the default `max_boundary_violations = 0` means a rule you bothered to declare is one you actually keep.
258260

261+
This repo's own `thresholds.toml` currently declares two: the one above, plus `crates/calm-server/src/watcher.rs``crates/calm-server/src/tools/` ("the background reindex/watch loop must not depend on the MCP tool-handler layer it runs independently of") — both hold at 0 violations.
262+
259263
## Deployment
260264

261265
- `cargo build --release` → static (musl on Linux) binaries via `.github/workflows/release.yml`, 5-target matrix with `SHA256SUMS` + build-provenance attestation for every asset: `x86_64-unknown-linux-musl`, `aarch64-unknown-linux-musl`, `aarch64-apple-darwin`, `x86_64-apple-darwin`, `x86_64-pc-windows-msvc`. `scripts/mcp-launcher.sh`/`scripts/install.sh` download and checksum-verify the right platform's build automatically when checkout is on (or you're installing) a matching git tag.
@@ -277,7 +281,7 @@ cargo test --workspace # unit + integration (embeddings i
277281
cargo test --test parity_test test_formal_edges # Stack Graphs regression corpus
278282
```
279283

280-
Five CI jobs run on every PR: `verify` (fmt/clippy/test/audit), `stack-graphs-corpus` (formal-resolver parity), `embeddings` (clippy + test with the `embeddings` feature), `all-languages` (fixture-repo indexing across all 24 parsed languages), `js-client-interop` (cross-checks the tool schema against a real JS MCP SDK client, not just Rust's own).
284+
Eight CI jobs run on every PR: `verify` (fmt/clippy/test/audit), `stack-graphs-corpus` (formal-resolver parity), `embeddings` (clippy + test with the `embeddings` feature), `no-stack-graphs-formal` (clippy + test with `stack-graphs-formal` off — the only CI coverage of the `resolver::formal` stub that feature gate compiles to), `all-languages` (fixture-repo indexing across all 24 parsed languages, plus `lsp-overlay`), `js-client-interop` (cross-checks the tool schema against a real JS MCP SDK client, not just Rust's own), `otel-http-features` (clippy + test with the `otel`/`http` features, plus a guard against `opentelemetry` core version skew), `fitness-check` (runs `calm fitness-check` against this repo's own index — see [Fitness check](#fitness-check--the-ci-gate) below).
281285

282286
The full workspace suite — 1,000+ tests — passes clean, with a handful of `#[ignore]`d live-binary integration tests (e.g. `rust-analyzer`/`scip-go`/`scip-java`) that need external tools not installed in every environment.
283287

@@ -291,7 +295,7 @@ The full workspace suite — 1,000+ tests — passes clean, with a handful of `#
291295
- [`docs/mcp-client-setup.md`](docs/mcp-client-setup.md) — every MCP client install path in detail, including Windsurf/Devin Desktop and Codex global config.
292296
- [`docs/http-transport.md`](docs/http-transport.md) — the opt-in remote/HTTP transport (`calm serve --http`): loopback-by-default, the fail-closed `--allow-remote` + token requirement, why remote exposure forces a read-only preset, and the TLS/reverse-proxy expectation.
293297
- [`AGENTS.md`](AGENTS.md) — the full tool-by-tool workflow guide this project's own agents follow.
294-
- [`benchmarks/`](benchmarks/) — the measurement suite behind every number in this README: `b2_call_graph_quality/` (precision/recall vs. a SCIP oracle), `b4_token_efficiency/` (token cost vs. a naive baseline, per task), `b11_extended_competitor_ab/` (real calls against 4 other live MCP servers, not self-reported numbers), `resolution/` (tier-distribution baseline across 19 real OSS repos, one per language). Unflattering results are published alongside good ones on purpose — `benchmarks/README.md` states that policy.
298+
- [`benchmarks/`](benchmarks/) — the measurement suite behind every number in this README, and a few more: `b2_call_graph_quality/` (precision/recall vs. a SCIP oracle), `b3_search_quality/` (hybrid RRF vs. FTS-only vs. raw grep, NDCG@10), `b4_token_efficiency/` (token cost vs. a naive baseline, per task), `b6_tool_call_efficiency/` (round-trips: naive multi-call vs. one MCP call), `b7_task_correctness/` (real rename refactors across 6 language corpora — fd/Rust, flask/Python, express/JS, zod/TS, gin/Go, spring-petclinic/Java — checked against an independent pass/fail oracle, not an LLM judge), `b11_extended_competitor_ab/` (real calls against 4 other live MCP servers, not self-reported numbers), `b12_tier1_tier2_tool_correctness/` (9 tools driven live over JSON-RPC against 6 external OSS repos, ground-truthed against regex/`git grep`), `resolution/` (tier-distribution baseline across 19 real OSS repos, one per language). Unflattering results are published alongside good ones on purpose — `benchmarks/README.md` states that policy.
295299

296300
## License
297301

0 commit comments

Comments
 (0)