Skip to content

feat(score): add Rust profile (#[test] / tokio::test / rstest / proptest) - #5

Open
hossein-webdev wants to merge 1 commit into
rollinsio:mainfrom
hossein-webdev:add-rust-profile
Open

hossein-webdev wants to merge 1 commit into
rollinsio:mainfrom
hossein-webdev:add-rust-profile

Conversation

@hossein-webdev

Copy link
Copy Markdown

Adds a rust profile to scripts/score.py, following the existing per-language dict pattern, plus 13 named regression tests. Rust felt like a conspicuous gap in a benchmark about test quality — it has strong idioms for every axis you measure.

Calibrated against three real, well-tested suites in the same spirit as the kotlin/swift work: serde_json, rust-lang/regex, and RustCrypto/hashes.

Rust-specific findings

Each of these is pinned by a test, because each one silently wrecked the numbers before I caught it:

  1. Inline #[cfg(test)] mod tests is Rust's dominant unit-test idiom. A tests/-only file pattern scores those suites as empty, so test_file accepts .rs generally — the same permissiveness the python profile already has.
  2. #[cfg(test)] must not count as a test definition, or every inline test module inflates test_count by one and flatters the D.1/D.2 denominators. The pattern requires test (or ns::test) immediately after #[, which cfg( doesn't satisfy.
  3. rustfmt splits long asserts across lines — and long literals are exactly the ones that get split. A line-anchored B.1 scored serde_json's suite at 12 against a hand count of 36. The expected-last branch now scans [^;]{0,200}?, which crosses newlines but can't cross a statement boundary. (Same shape as the note in your go B.1 about table-driven suites scoring ~0.)
  4. Raw strings r#"…"# are the embedded-JSON fixture idiom and legitimately contain ", so a [^"\n] class stops at the first inner quote and scores serde_json's JSON vectors at zero. They get their own branch.
  5. Numeric crates park hex vectors inside tuples and slicesassert_eq!(big.hi64(), (0xA000000000000000, false)), from_u32(&[0x00140000, 0x140000]) — so anchoring hex to the comma missed them.

Two deliberate design calls, both matching existing precedent:

  • A.2 is None (uncountable), as in Go and Swift. A #[cfg(test)] module sees private items by design, and integration tests under tests/ can only reach pub — there's no countable private-access smell, and returning 0 would hand the axis a free win.
  • rstest's bare #[case] — the marker for which function argument receives the row — is excluded from param; only #[case(...)] rows count. Counting both gave every parametrized test one extra. (My own test caught this.)

Accuracy

Heuristic, same tier as the other non-Python profiles. Measured against a hand count on the calibration corpus:

suite scorer B.1 hand count
rust-lang/regex 3 3
serde_json 41 36

serde_json over-counts by 5 — the bounded scan occasionally picks up a hex literal from an adjacent statement. Trustworthy for trends and worst-offenders, as your docs frame the non-validated profiles; "validated": False is set accordingly.

Testing

python -m pytest tests/ -q85 passed, 4 xfailed (the 4 xfails are the pre-existing ones, untouched).

Counts in the new tests are hand-derived by reading each fixture, not recomputed with the profile's own regex.

Also updated

--lang lists in SKILL.md and README.md, and the module docstring's supported-language block and calibration note.


Unrelated, and happy to send it separately if useful: render() emits a that raises UnicodeEncodeError on Windows consoles using cp1252 (--json is unaffected). Repro: python score.py --tests <dir> on Windows without PYTHONIOENCODING=utf-8.

Thanks for publishing this — the rubric is the most useful thing I've read on test quality in a while, and it's now the basis of a test-quality skill in an MIT skill pack I maintain, credited to you in the README and the skill itself.

…t / proptest)

Adds a 
ust language profile to score.py following the existing per-language dict pattern, plus 13 named regression tests.

Calibrated against three real, well-tested suites - serde_json, rust-lang/regex and RustCrypto/hashes - in the same spirit as the kotlin/swift profiles.

Rust-specific calibration findings, each pinned by a test:
- Inline #[cfg(test)] mod tests under src/ is Rust's dominant unit-test idiom, so the test_file pattern accepts .rs generally (a tests/-only pattern scores those suites empty).
- #[cfg(test)] must not count as a test definition or every inline module inflates test_count.
- rustfmt splits long asserts across lines - exactly the ones holding long literals - so B.1 uses a bounded statement-scoped scan rather than a line-anchored one.
- Raw strings 
#\\...\\# are the embedded-JSON fixture idiom and legitimately contain quotes, so they need their own branch.
- Numeric crates park hex vectors inside tuples/slices, so B.1 matches hex anywhere within the assert.
- A.2 is None (uncountable) as in Go/Swift: cfg(test) modules see private items by design.
- rstest's bare #[case] argument marker is excluded from the parametrization count; only #[case(...)] rows count.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant