Skip to content

Commit 9f038a5

Browse files
committed
DESIGN: N8 updated with Python benchmark + sweep findings
1 parent 1416069 commit 9f038a5

1 file changed

Lines changed: 14 additions & 4 deletions

File tree

DESIGN.md

Lines changed: 14 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -277,10 +277,20 @@ vendor/opengrep # pinned engine (git submodule @ 8ee180dc)
277277
- **N1 — Reuse precision.** The "call this" gate: thresholds, containment weighting,
278278
`is_public_api`/visibility filtering, callability ranking.
279279
- **N5 — Rust-specific granularity.** Trait/impl/generic modeling beyond flat units.
280-
- **N8 — Per-language tuning.** Defaults are tuned on a Rust-only benchmark (D1
281-
prioritizes Rust anyway). To tune another language: extend the miner's keyword
282-
list, mine multi-version packages from npm/pypi caches, re-run `tests/sweep.sh`.
283-
The 4-language sanity suite is the regression floor meanwhile.
280+
- **N8 — Per-language tuning.** Partially answered (2026-06-11): the Python
281+
benchmark exists (`tests/mine-pypi.sh`, 28 package-versions, 11.4k pairs; the
282+
miner's token normalizer is language-aware via per-extension keyword tables).
283+
Baseline with Rust-tuned defaults is already strong (evolved MRR 0.86 vs
284+
Rust's 0.72 — fewer types means version edits perturb less); judge
285+
correlations mirror Rust (n=148: structure +0.78, purpose +0.64). **Sweep
286+
verdict: optimal channels are per-language.** Rust: `call_names+int_values`
287+
(strings cost renamed −5pp). Python: those PLUS `string_values+field_names`
288+
— ALL MRR 0.955→0.969, evolved MRR 0.862→0.929, samefile FPR(max-gate)
289+
0.068→0.036; `ty_descrs` actively hurts (annotation drift breaks even exact
290+
pairs). Depth 1 optimal/flat in both. Remaining (issue #1): wire
291+
per-language base configs into extraction; kernel/C benchmark; JS/Go.
292+
Python-specific: unittest↔pytest assert migrations dominate the evolved
293+
tail — normalization pass endorsed.
284294
- *(resolved: N2 → D7, N6 → D11, N7 → closure grafting)*
285295

286296
## Known levers, deliberately not pulled yet

0 commit comments

Comments
 (0)