Skip to content

Commit 3f89c3e

Browse files
rollinsioclaude
andauthored
Add CI, Kotlin/Swift dashboard, and repo-consistency cleanups (#4)
- Add a GitHub Actions workflow running all three pytest suites (benchmark tooling + both bundled skills) plus the sabotage check. - Generate the missing Kotlin/Swift interactive dashboard (docs/kotlin-swift-results.html) from the committed scorecard, with a static preview in examples/ and README links alongside the other two. - Drop the Kotlin/Swift repos from score_cross_language.py: the committed matrix was scored by the manifest-driven, module-scoped score_kotlin_swift_matrix.py, and rescoring them here (whole-repo, unscoped) would merge contradictory baselines into the cross-language scorecard. - Correct the CHANGELOG claim that the C.1 instruments "now agree" — score.py and aggregate_results.py still match different textual forms of the stdlib mock.patch idiom. - Frame reports/ as the coverage-driven control up front, so its mostly-"Worse" verdict matrix reads as the motivation for the quality experiment rather than a contradiction of it. - Move the orphaned baseline_summary.md into reports/ and link it; clarify README's 185→59 A.2 claim (coverage-arm vs quality-arm, not vs baseline); fix SKILL.md's two stale python|js|go-only mentions; mark gen_prompts.py and verify_run.sh as historical coverage-control tooling. Claude-Session: https://claude.ai/code/session_011BDzS3SyE6Sh3GE4nnSoWN Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent 673d7c6 commit 3f89c3e

12 files changed

Lines changed: 185 additions & 20 deletions

File tree

.claude/skills/test-quality/SKILL.md

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -34,7 +34,8 @@ model, drove the gain). The two reference docs are that experiment's distilled
3434
output:
3535
- `references/quality-contract.md` — 10 anti-fragility rules, each with the repair.
3636
- `references/scorecard.md` — the scoring axes, the improvement gate, the stop condition.
37-
- `scripts/score.py` — measures the auto-countable axes for any pytest suite.
37+
- `scripts/score.py` — measures the auto-countable axes for a suite in any
38+
supported language (Python/pytest, JS/TS, Go, Kotlin, Swift).
3839

3940
**Read both reference docs before starting.** They are the substance; this file
4041
is the procedure.
@@ -76,7 +77,7 @@ Identify the source package and its tests dir. Detect the **language/framework**
7677
### 2. Capture the baseline (do this BEFORE changing anything)
7778
- Run the suite with branch coverage; record line % and branch % — this is the
7879
coverage floor.
79-
- `python <skill>/scripts/score.py --tests <tests_dir> [--lang python|js|go]`
80+
- `python <skill>/scripts/score.py --tests <tests_dir> [--lang python|js|go|kotlin|swift]`
8081
for the auto axes (`--lang` auto-detects if omitted).
8182
- Preserve the starting state so you can score against it: copy the tests dir
8283
aside, or note the git ref. Later runs use `--baseline <that copy>`.

.github/workflows/ci.yml

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
name: CI
2+
3+
on:
4+
push:
5+
branches: [main]
6+
pull_request:
7+
8+
jobs:
9+
test:
10+
runs-on: ubuntu-latest
11+
steps:
12+
- uses: actions/checkout@v4
13+
- uses: actions/setup-python@v5
14+
with:
15+
python-version: "3.12"
16+
- name: Install measurement-tooling deps
17+
run: pip install -r requirements-dev.txt
18+
- name: Unit tests (benchmark tooling + both skills)
19+
run: >
20+
python -m pytest -q
21+
tests/
22+
.claude/skills/test-quality/tests/
23+
.claude/skills/results-dashboard/tests/
24+
- name: Sabotage check (curated mutations must all be caught)
25+
run: python scripts/sabotage_check.py

CHANGELOG.md

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -35,8 +35,11 @@ the pre-fix scorer. Each fix has a named regression test in
3535
paren hid the `match=` kwarg.
3636
- **Python C.1 missed the stdlib `mock.patch` idiom** — `from unittest import
3737
mock; mock.patch(...)` / `mock.patch.object(...)` scored 0 (only `mocker`,
38-
bare `patch(`, `Mock(`, `MagicMock` counted). `aggregate_results.py` already
39-
counted `unittest.mock` — the instruments now agree.
38+
bare `patch(`, `Mock(`, `MagicMock` counted). `score.py` now catches it via
39+
a `mock.patch` alternative. Note the instruments still differ textually:
40+
`aggregate_results.py` matches the literal `unittest.mock` (which does not
41+
fire on `from unittest import mock` + `mock.patch(...)`) and lacks the
42+
`mock.patch` alternative, so the two can disagree on that idiom.
4043
- **Go `test_def` missed testify suite methods and counted `TestMain`** — now
4144
matches an optional method receiver (`func (s *Suite) TestFoo(`) and
4245
excludes exactly `TestMain` (a `TestMainPage` still counts).

README.md

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -92,7 +92,8 @@ Across **27 quality-driven runs** (9 libraries × 3 iteration budgets),
9292
structural: the generated suites are dramatically **more LOC-efficient**
9393
than every baseline, carry **zero substring-match assertions** and **no
9494
hand mocks** of the code under test, cut private-symbol access sharply
95-
(185 → 59 across the Python matrix — though not to zero), and are more
95+
(the coverage-driven arms' 185 total fell to 59 under quality-driven
96+
across the Python matrix — though not to zero), and are more
9697
parametrized than most baselines.
9798

9899
Three honest caveats the numbers also surfaced:
@@ -122,13 +123,18 @@ Three honest caveats the numbers also surfaced:
122123
- [`docs/python-results.html`](docs/python-results.html) — the
123124
Python dashboard, including the decomposition of *why* coverage-driven
124125
scored 2/9 and quality-driven scored 9/9.
126+
- [`docs/kotlin-swift-results.html`](docs/kotlin-swift-results.html)
127+
the Kotlin/Swift matrix dashboard: six libraries × three policies,
128+
all 18 arms vs their maintainer-written baselines.
125129
- [`FINDINGS.md`](FINDINGS.md) — the full running analysis.
126130

127131
Static previews live in [`examples/`](examples/) (GitHub renders the HTML
128132
dashboards as source, so open them locally for the interactive charts):
129133

130134
[![Cross-language results — 18/18 arms beat the human baseline](examples/cross-language-results.png)](docs/cross-language-results.html)
131135

136+
[![Kotlin + Swift matrix — all 18 arms beat their maintainer-written baselines](examples/kotlin-swift-results.png)](docs/kotlin-swift-results.html)
137+
132138
## The quality scorecard
133139

134140
The criteria above are a multi-axis rubric; the scorer auto-counts the
@@ -195,7 +201,8 @@ green** — the three reds are all *oneshot* (one pass, no repair) and fail to
195201
green" pattern the Python/JS runs showed. Every W/L/T was re-scored independently
196202
of the generating agent, and every green count re-run from the real toolchain.
197203
The two largest repos (`kotlinx.serialization`, `swift-collections`) are scoped to
198-
a coherent core module, stated explicitly. See
204+
a coherent core module, stated explicitly. See the
205+
[interactive dashboard](docs/kotlin-swift-results.html),
199206
[`results-kotlin-swift-scorecard.md`](results-kotlin-swift-scorecard.md) and
200207
[`reports/kotlin-swift-generation.md`](reports/kotlin-swift-generation.md).
201208
(These profiles are heuristic, not the empirically-validated Python set, so read

0 commit comments

Comments
 (0)