You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs(readme): refresh stale measured numbers and undercounted lists (#49)
- Bump the "Proof, not promises" table and repo_overview example to the
current 237 files / 3,583 symbols, hub/dead-code/edge-coverage/complexity
percentages, and add the avg_distance (Martin/OOD) row that was missing.
- Fitness check now measures 11 metrics, not 10 (avg_distance was added
without updating the table).
- Note that thresholds.toml already declares two [[boundaries]] rules, not
just the one shown as an example.
- Testing section said "Five CI jobs" — ci.yml actually runs eight
(no-stack-graphs-formal, otel-http-features, and fitness-check were
undocumented).
- Further reading's benchmark list was missing four already-Implemented
benchmarks (b3, b6, b7, b12).
Co-authored-by: Eilodon <noreply@anthropic.com>
| Architecture boundary violations (`boundary_violations`) | 0 (gate: ≤ 0) — the `watcher → tools` import previously flagged here was fixed by relocating the shared `RwLockExt`/`LockExt` traits it needed out of `tools/common.rs` into their own `sync_ext` module |
142
143
| Token efficiency vs. a naive read-the-files baseline |`source`**241x** · `edit_context`**193x** · `locate`**29x** · `callers`**1.0x** — median 111x across the four benchmark tasks ([methodology](benchmarks/b4_token_efficiency/)) |
@@ -226,7 +227,7 @@ Distinct from the `tools` above — MCP Prompts (`prompts/list`, `prompts/get`)
226
227
227
228
Run for real in `.github/workflows/ci.yml`'s `fitness-check` job on every push/PR — `calm index` first (a fresh checkout has no `.calm/index.db` yet), then `calm fitness-check --project-root . --config thresholds.toml`. That `--config` flag is not optional: without it, `[[boundaries]]` and `[config_drift]` are silently treated as "no rules declared" rather than erroring — only the numeric thresholds have a real default.
228
229
229
-
`calm fitness-check` measures 10 metrics against thresholds declared in `thresholds.toml`:
230
+
`calm fitness-check` measures 11 metrics against thresholds declared in `thresholds.toml`:
230
231
231
232
| Metric | What it measures | Default threshold |
232
233
|---|---|---|
@@ -237,6 +238,7 @@ Run for real in `.github/workflows/ci.yml`'s `fitness-check` job on every push/P
237
238
|`hotspot_risk`| Highest hotspot score in the codebase | ≤ 0.75 |
238
239
|`edge_coverage_pct`| % of symbols with at least one call edge | ≥ 60% |
239
240
|`high_complexity_pct`| % of functions/methods with McCabe cyclomatic complexity > 10 (AST-based; Tier-0.5 languages always report complexity 1) | ≤ 15.0% |
241
+
|`avg_distance`| Martin/OOD average distance from the main sequence — how far each file's abstractness sits from the ideal implied by its instability (Ca/Ce) | ≤ 1.00 |
240
242
|`boundary_violations`| Count of `import_edges` violating a declared `[[boundaries]]` rule | ≤ 0 |
241
243
|`boundary_ambiguous_count`| Count of symbols with an ambiguous line boundary (shared with a neighbor) — `edit_symbol` replace on these is refused until resolved | ≤ 0 |
242
244
|`config_drift_count`| Count of doc file-path references (declared via `[config_drift].doc_paths`) pointing at nothing real | ≤ 0 |
@@ -256,6 +258,8 @@ reason = "indexer (extraction) must stay upstream of analysis (dead-code, hotspo
256
258
257
259
`calm fitness-check` reports each violation concretely (the real from/to path, the rule, and the reason) outside `--json` mode; the default `max_boundary_violations = 0` means a rule you bothered to declare is one you actually keep.
258
260
261
+
This repo's own `thresholds.toml` currently declares two: the one above, plus `crates/calm-server/src/watcher.rs` → `crates/calm-server/src/tools/` ("the background reindex/watch loop must not depend on the MCP tool-handler layer it runs independently of") — both hold at 0 violations.
262
+
259
263
## Deployment
260
264
261
265
-`cargo build --release` → static (musl on Linux) binaries via `.github/workflows/release.yml`, 5-target matrix with `SHA256SUMS` + build-provenance attestation for every asset: `x86_64-unknown-linux-musl`, `aarch64-unknown-linux-musl`, `aarch64-apple-darwin`, `x86_64-apple-darwin`, `x86_64-pc-windows-msvc`. `scripts/mcp-launcher.sh`/`scripts/install.sh` download and checksum-verify the right platform's build automatically when checkout is on (or you're installing) a matching git tag.
@@ -277,7 +281,7 @@ cargo test --workspace # unit + integration (embeddings i
277
281
cargo test --test parity_test test_formal_edges # Stack Graphs regression corpus
278
282
```
279
283
280
-
Five CI jobs run on every PR: `verify` (fmt/clippy/test/audit), `stack-graphs-corpus` (formal-resolver parity), `embeddings` (clippy + test with the `embeddings` feature), `all-languages` (fixture-repo indexing across all 24 parsed languages), `js-client-interop` (cross-checks the tool schema against a real JS MCP SDK client, not just Rust's own).
284
+
Eight CI jobs run on every PR: `verify` (fmt/clippy/test/audit), `stack-graphs-corpus` (formal-resolver parity), `embeddings` (clippy + test with the `embeddings` feature), `no-stack-graphs-formal` (clippy + test with `stack-graphs-formal` off — the only CI coverage of the `resolver::formal` stub that feature gate compiles to), `all-languages` (fixture-repo indexing across all 24 parsed languages, plus `lsp-overlay`), `js-client-interop` (cross-checks the tool schema against a real JS MCP SDK client, not just Rust's own), `otel-http-features` (clippy + test with the `otel`/`http` features, plus a guard against `opentelemetry` core version skew), `fitness-check` (runs `calm fitness-check` against this repo's own index — see [Fitness check](#fitness-check--the-ci-gate) below).
281
285
282
286
The full workspace suite — 1,000+ tests — passes clean, with a handful of `#[ignore]`d live-binary integration tests (e.g. `rust-analyzer`/`scip-go`/`scip-java`) that need external tools not installed in every environment.
283
287
@@ -291,7 +295,7 @@ The full workspace suite — 1,000+ tests — passes clean, with a handful of `#
291
295
-[`docs/mcp-client-setup.md`](docs/mcp-client-setup.md) — every MCP client install path in detail, including Windsurf/Devin Desktop and Codex global config.
292
296
-[`docs/http-transport.md`](docs/http-transport.md) — the opt-in remote/HTTP transport (`calm serve --http`): loopback-by-default, the fail-closed `--allow-remote` + token requirement, why remote exposure forces a read-only preset, and the TLS/reverse-proxy expectation.
293
297
-[`AGENTS.md`](AGENTS.md) — the full tool-by-tool workflow guide this project's own agents follow.
294
-
-[`benchmarks/`](benchmarks/) — the measurement suite behind every number in this README: `b2_call_graph_quality/` (precision/recall vs. a SCIP oracle), `b4_token_efficiency/` (token cost vs. a naive baseline, per task), `b11_extended_competitor_ab/` (real calls against 4 other live MCP servers, not self-reported numbers), `resolution/` (tier-distribution baseline across 19 real OSS repos, one per language). Unflattering results are published alongside good ones on purpose — `benchmarks/README.md` states that policy.
298
+
- [`benchmarks/`](benchmarks/) — the measurement suite behind every number in this README, and a few more: `b2_call_graph_quality/` (precision/recall vs. a SCIP oracle), `b3_search_quality/` (hybrid RRF vs. FTS-only vs. raw grep, NDCG@10), `b4_token_efficiency/` (token cost vs. a naive baseline, per task), `b6_tool_call_efficiency/` (round-trips: naive multi-call vs. one MCP call), `b7_task_correctness/` (real rename refactors across 6 language corpora — fd/Rust, flask/Python, express/JS, zod/TS, gin/Go, spring-petclinic/Java — checked against an independent pass/fail oracle, not an LLM judge), `b11_extended_competitor_ab/` (real calls against 4 other live MCP servers, not self-reported numbers), `b12_tier1_tier2_tool_correctness/` (9 tools driven live over JSON-RPC against 6 external OSS repos, ground-truthed against regex/`git grep`), `resolution/` (tier-distribution baseline across 19 real OSS repos, one per language). Unflattering results are published alongside good ones on purpose — `benchmarks/README.md` states that policy.
0 commit comments