feat(score): add Rust profile (#[test] / tokio::test / rstest / proptest) - #5
Open
hossein-webdev wants to merge 1 commit into
Open
hossein-webdev wants to merge 1 commit into
hossein-webdev wants to merge 1 commit into
Conversation
…t / proptest) Adds a ust language profile to score.py following the existing per-language dict pattern, plus 13 named regression tests. Calibrated against three real, well-tested suites - serde_json, rust-lang/regex and RustCrypto/hashes - in the same spirit as the kotlin/swift profiles. Rust-specific calibration findings, each pinned by a test: - Inline #[cfg(test)] mod tests under src/ is Rust's dominant unit-test idiom, so the test_file pattern accepts .rs generally (a tests/-only pattern scores those suites empty). - #[cfg(test)] must not count as a test definition or every inline module inflates test_count. - rustfmt splits long asserts across lines - exactly the ones holding long literals - so B.1 uses a bounded statement-scoped scan rather than a line-anchored one. - Raw strings #\\...\\# are the embedded-JSON fixture idiom and legitimately contain quotes, so they need their own branch. - Numeric crates park hex vectors inside tuples/slices, so B.1 matches hex anywhere within the assert. - A.2 is None (uncountable) as in Go/Swift: cfg(test) modules see private items by design. - rstest's bare #[case] argument marker is excluded from the parametrization count; only #[case(...)] rows count.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
rustprofile toscripts/score.py, following the existing per-language dict pattern, plus 13 named regression tests. Rust felt like a conspicuous gap in a benchmark about test quality — it has strong idioms for every axis you measure.Calibrated against three real, well-tested suites in the same spirit as the kotlin/swift work: serde_json, rust-lang/regex, and RustCrypto/hashes.
Rust-specific findings
Each of these is pinned by a test, because each one silently wrecked the numbers before I caught it:
#[cfg(test)] mod testsis Rust's dominant unit-test idiom. Atests/-only file pattern scores those suites as empty, sotest_fileaccepts.rsgenerally — the same permissiveness the python profile already has.#[cfg(test)]must not count as a test definition, or every inline test module inflatestest_countby one and flatters the D.1/D.2 denominators. The pattern requirestest(orns::test) immediately after#[, whichcfg(doesn't satisfy.[^;]{0,200}?, which crosses newlines but can't cross a statement boundary. (Same shape as the note in yourgoB.1 about table-driven suites scoring ~0.)r#"…"#are the embedded-JSON fixture idiom and legitimately contain", so a[^"\n]class stops at the first inner quote and scores serde_json's JSON vectors at zero. They get their own branch.assert_eq!(big.hi64(), (0xA000000000000000, false)),from_u32(&[0x00140000, 0x140000])— so anchoring hex to the comma missed them.Two deliberate design calls, both matching existing precedent:
None(uncountable), as in Go and Swift. A#[cfg(test)]module sees private items by design, and integration tests undertests/can only reachpub— there's no countable private-access smell, and returning 0 would hand the axis a free win.#[case]— the marker for which function argument receives the row — is excluded fromparam; only#[case(...)]rows count. Counting both gave every parametrized test one extra. (My own test caught this.)Accuracy
Heuristic, same tier as the other non-Python profiles. Measured against a hand count on the calibration corpus:
serde_json over-counts by 5 — the bounded scan occasionally picks up a hex literal from an adjacent statement. Trustworthy for trends and worst-offenders, as your docs frame the non-validated profiles;
"validated": Falseis set accordingly.Testing
python -m pytest tests/ -q→ 85 passed, 4 xfailed (the 4 xfails are the pre-existing ones, untouched).Counts in the new tests are hand-derived by reading each fixture, not recomputed with the profile's own regex.
Also updated
--langlists inSKILL.mdandREADME.md, and the module docstring's supported-language block and calibration note.Unrelated, and happy to send it separately if useful:
render()emits a↓that raisesUnicodeEncodeErroron Windows consoles using cp1252 (--jsonis unaffected). Repro:python score.py --tests <dir>on Windows withoutPYTHONIOENCODING=utf-8.Thanks for publishing this — the rubric is the most useful thing I've read on test quality in a while, and it's now the basis of a test-quality skill in an MIT skill pack I maintain, credited to you in the README and the skill itself.