CHANGELOG.md

Changelog

Tracks changes to the experiment design, prompts, and tooling between experiments. Each version section is the design that produced (or will produce) that experiment's worktrees. Once an experiment executes, its prompts and metadata should not be edited — subsequent changes go into the next section.

Format roughly follows Keep a Changelog.

---

[Unreleased]

Changes intended for a future round go here. Motivated by the cross-language Finding 17 (B.1).

Fixed — scorer (2026-07 correctness review)

A correctness review of score.py (the live instrument the cross-language and Kotlin/Swift scripts import — the Python experiment's score_quality.py carries its own frozen copies of the regexes and is untouched). A re-score of preserved worktrees with the fixed scorer may therefore shift individual counts slightly; the committed results-*.json remain the record produced by the pre-fix scorer. Each fix has a named regression test in tests/test_score.py.

• Python A.2 counted every snake_case import as private access — the name-list in the from … import … alternative could consume the head of a public name (from requests.utils import super_len counted). The _ is now anchored to the start of an identifier, and the name-list no longer spans newlines (consecutive import lines used to merge into one match and undercount).
• Python A.1 missed pytest.raises((A, B), match=…) — the tuple's closing paren hid the match= kwarg.
• Python C.1 missed the stdlib mock.patch idiom — from unittest import mock; mock.patch(...) / mock.patch.object(...) scored 0 (only mocker, bare patch(, Mock(, MagicMock counted). score.py now catches it via a mock.patch alternative. Note the instruments still differ textually: aggregate_results.py matches the literal unittest.mock (which does not fire on from unittest import mock + mock.patch(...)) and lacks the mock.patch alternative, so the two can disagree on that idiom.
• Go test_def missed testify suite methods and counted TestMain — now matches an optional method receiver (func (s *Suite) TestFoo() and excludes exactly TestMain (a TestMainPage still counts).
• Test-file patterns matched ancestor directories of --tests — a repo under e.g. /home/ci/latest/ turned every .kt source file into a "test" (latest/ ends in test), poisoning counts and detect_lang. Patterns now match the path relative to the target (keeping the target dir's own name — the express test/ layout still detects).
• An empty/unrecognized suite could "beat" a real baseline — zero tests trivially wins every lower-better count axis, and D1 = 0.0 (LOC/test) read as best-possible. D.1/D.2 are now N/A when no tests are recognized, and the rendered tally refuses to report BETTER for a zero-test suite.
• Harness scripts no longer clobber committed results when clones are absent — scorer_check.py, score_cross_language.py, and score_kotlin_swift_matrix.py used to overwrite results-*.{json,md} with empty payloads when run without the (git-ignored) library clones on disk; they now refuse and exit non-zero. score_cross_language.py also no longer crashes (KeyError) rendering markdown when a listed repo is un-cloned.

Known gaps — scorer (pinned as strict xfail in tests/test_score.py)

• Swift Testing @Test func testFoo() counts as 2 tests (matches both test_def alternatives), deflating D.1.
• Chai's two-arg expect(fn).to.throw(TypeError, 'substr') — a message substring assert — is not counted as A.1.
• test_def is a plain grep: def test_…( inside comments or string literals inflates test_count (flatters the D.1/D.2 denominators).
• Python B.1's charset has no space, so human-readable fixed vectors (== "Hello, world …") don't count — unlike the any-12+-char rule in JS/Kotlin/Swift. Fold into the B.1 harmonization below.

Proposed — scorecard (measurement)

• Re-shape B.1 from an absolute count to a per-test ratio (fixed-vectors / test, like D.2). The cross-language experiment showed every arm's only loss was B.1, and that an absolute B.1 fights D.1 (LOC efficiency): a clean small suite cannot match a 1941-test baseline's 914 fixed vectors without padding that regresses D.1. A ratio removes the size-scaling and the D.1 conflict.
• Extend the JS B.1 profile to count framework-matcher fixed vectors — supertest .expect(status, body), it.each table rows, object/array deep-equals — which the current inline-string-literal regex misses (so fixed-vector-dense oneshot suites read B.1 ≈ 3–4). NOT applied mid-cross-language to avoid moving the frozen instrument post-generation.
• Harmonize B.1 across languages (fold in with the ratio change): the minimum-literal thresholds disagree (Python 16, JS/Kotlin/Swift 12, Go 12 or 8 depending on the alternative), and Go's bare []byte(" alternative counts input construction — a generated Go suite can pump B.1 with literals that assert nothing.
• Reconcile the Go A.1/B.1 split with the other profiles: Go counts exact message equality (err.Error() == "…", assert.EqualError) as the A.1 smell, while the JS/Kotlin/Swift profiles deliberately classify exact equality as a B.1 fixed vector and reserve A.1 for partial matchers. The same idiom is currently penalized in Go and rewarded in JS.
• Decide the Kotlin C.1 unit: every every { … }/coEvery { … } stub line counts, which is the per-configuration double-counting the JS profile documents avoiding (mockReturnValue is excluded there). Deliberate at calibration time (pinned in the kotlin-result regression) but inconsistent across profiles.

Added — scorer (languages: Kotlin + Swift)

• kotlin and swift profiles in score.py, applying the same axes as the other languages with framework-appropriate regexes:
  • kotlin — kotlin.test / JUnit5 / Kotest (.kt). Notable calibration choices: @Test-family annotations drive test_def (the @Test\b boundary excludes @TestInstance-style config); Kotest leaves count only when written name("…") { … } with a body (so a bare local-helper call like test("Z") does not inflate the count — the kotlinx-datetime regression); A.1 is partial message-matchers only (an assertTrue(x.message is T) type-check is excluded — the kotlinx.serialization regression); B.1 counts triple-quoted """…""" vectors and the expected = … named arg; A.2 counts reflection into privates.
  • swift — XCTest / Swift Testing / Quick+Nimble (.swift). test_def spans func test…, @Test, and it("…"); param is Swift Testing @Test(arguments:); B.1 also counts Apple's StdlibUnittest expectEqual helper and #expect(x == "…"); A.2 is n/a (@testable import of internal is idiomatic, private unreachable — as in Go).
• Calibrated against six real, well-tested suites — kotlinx.serialization, kotlinx-datetime, kotlin-result; swift-argument-parser, swift-collections, SwiftyJSON. Every calibration finding is a named regression in tests/test_score.py (24 new cases); baselines recorded in reports/kotlin-swift-baselines.md. Tier: heuristic (not the empirically- validated Python tier).
• Cross-language harness extended with the six targets in setup_cross_language.py and score_cross_language.py (+.gitignore clone dirs). Both scripts skip any target whose <repo>/base clone is absent, so the committed JS/Go run is undisturbed.

Executed — Kotlin + Swift generation matrix (full 6×3 = 18 arms)

• Ran the delete-and-regenerate loop across all six repos × three iteration policies (oneshot / iter2 / iter20), built/run with the real toolchains (JDK 17 + each repo's Gradle wrapper; Apple Swift 6.1.2). The two largest repos (kotlinx.serialization → JSON module; swift-collections → OrderedCollections) are scoped to a coherent core module; baseline is each repo's whole human suite.
• Result: 18/18 arms beat the human baseline on the countable axes; 15/18 green. The three reds are all oneshot and are compile failures (a private type in @Test(arguments:); mutating a let; a JsonObject/Map type-inference site) left unrepaired per the one-pass policy — iter2 fixes each in one round. This reproduces the Python/JS "one pass wins the quality axes but ships failures; iteration makes it green" finding in Kotlin and Swift. Iteration also deepens quality (e.g. swift-collections B.1 3→31→45 across oneshot→iter2→iter20, flipping the B.1 loss to a win).
• Integrity: every W/L/T was re-scored by the orchestrator independently of the generating agent, and every green count re-run from the real toolchain; one apparent green (a stale verification-probe XML) was caught and corrected to red on re-run.
• Scores in results-kotlin-swift-scorecard.{json,md} (the {baselines, arms} shape, via scripts/score_kotlin_swift_matrix.py over bench-clones/.matrix-manifest.json); narrative in reports/kotlin-swift-generation.md; the decisive (iter20) suite for each repo is preserved under reports/generated-suites/.
• Caveat: the Kotlin/Swift profiles are heuristic. The recurring B.1 loss on the large repos is the absolute-count-vs-suite-size artifact this CHANGELOG's [Unreleased] B.1-as-ratio proposal targets — the scoped arms are compared against a much larger whole-repo baseline.

---

[Cross-language (JS/TS + Go)] — 2026-05-29 (executed)

Cross-language extension: does scorecard-anchored prompting generalize beyond Python? Six repos (JS/TS: express, jsonwebtoken, zod; Go: chi, gjson, golang-jwt) × oneshot/iter2/iter20 = 18 arms. Result: 18/18 beat baseline. See FINDINGS §15–19.

Model: Opus 4.8, held fixed from the quality experiment (per the quality note to stop moving the model once the prompt stabilized). Arms generated by autonomous subagents, one per worktree.

Added

• prompts/cross-language/ — language-parameterized prompt set (common_header, quality_contract, quality_scorecard, oneshot/iter2/iter20). Self-scoring is a single score.py --lang <L> --baseline call, not the quality experiment's grep/wc recipes.
• scripts/setup_cross_language.py — creates the 18 wt-r3-<policy> worktrees, wires deps (npm symlink, pnpm relink for zod's monorepo, shared cache for Go), and materializes .rex_prompt.md per worktree.
• scripts/score_cross_language.py — independent gen-vs-baseline recompute via the per-language profiles → results-cross-language-scorecard.{json,md}.
• scripts/scorer_check.py → results-scorer-check.{json,md} — baseline-only profile validation over the six clones (not a benchmark run).

Changed — scorer (measurement), calibrated before generation

• JS profile calibrated to Chai / node:assert / sinon. A.1/B.1 saw only Jest/Vitest matchers, so Mocha+Chai and node:assert suites read ~0. Now: B.1 counts Chai (.to.equal/.eql/.deep.equal) and node/chai assert.equal/strictEqual/deepEqual 12+ char literals (express 0→28, jsonwebtoken 0→21); A.1 restricted to partial matchers (exact == is a B.1 fixed vector, mirroring the Python match=/in str( split, removing the A.1∩B.1 double-count); C.2 recognizes sinon.useFakeTimers (so the JS C.1=0 readings are confirmed correct).
• Go B.1 calibrated to the table-driven idiom (surfaced by the gjson/oneshot pilot): counts comparison sites against want/expected-named fields/vars (got != tc.want) and named expected-field literals, not just inline want :=/== "lit"/[]byte(". gjson gen 0→20; committed before the 17-arm fan-out so iterative agents self-scored Go correctly.

Findings (FINDINGS.md §15–19)

• 18/18 arms beat baseline; D.1 and D.2 are universal wins.
• All 6 oneshot arms shipped red (1–4 self-authored failures, unrepaired per policy); all 12 iterative arms green — iteration's value is correctness.
• Every loss is B.1, an absolute count that scales with suite size and fights D.1 → motivates the [Unreleased] B.1-ratio proposal.
• Integrity: 18/18 clean (no git recovery, no baseline-test reading; spot-checked).

---

[Quality-driven (Python)] — 2026-05-28 (designed; not yet executed)

The defining shift from the coverage-driven baseline: the success criterion is winning a multi-axis quality scorecard, not hitting a coverage number. Coverage becomes a non-regression floor only.

Model: the quality experiment uses Claude Opus 4.8 (claude-opus-4-8). The coverage-driven baseline used Opus 4.7. A model change is a confound for any cross-experiment comparison; we accept it because (a) Opus 4.8 is the model we'd actually deploy now and (b) the coverage-driven baseline + the quality experiment differ on so many prompt axes that isolating the model effect from the prompt-design effect was never going to be clean anyway. Future experiments should hold the model fixed until the prompt has stabilized.

Added

• prompts/quality/quality_scorecard.md — multi-axis quality goal (axes A–F: anti-fragility counts, rigor signals, mocking footprint, LOC efficiency, suite correctness, coverage floor). The session computes a baseline scorecard at the start and races to beat it.
• prompts/quality/common_header.md — reframes coverage as a floor; forbids git-history recovery; forbids pip install -e . from worktrees; names framework primitives per repo.
• prompts/quality/quality_contract.md — 10 anti-fragility rules with negative + positive examples covering: error-message substring matching, recomputed crypto, private symbols, tautological constructor readbacks, or-joined assertions, hand-coded char sets, parametrize/fixtures/inheritance, boundary tests, REPL verification of stdlib assumptions, framework-primitive preference.
• prompts/quality/oneshot.md / iter2.md / iter20.md — per-policy bodies anchored on scorecard wins.
• runs/coverage.md — preserved coverage-driven control worktree inventory with verified pass/fail and coverage figures.
• FINDINGS.md — running log of findings; the coverage-driven control entry has 9 sub-findings.
• CHANGELOG.md — this file.
• configs/<repo>/bench.coveragerc — versioned copies of the coverage config used in the coverage-driven control (preserved so a future bootstrap can regenerate without depending on the OSS dirs).
• scripts/setup_quality.py — materializes wt-r2-* worktrees, copies bench.coveragerc, writes .rex_prompt.md and start.sh.
• scripts/launch_quality.sh — opens 9 iTerm tabs for the quality experiment.
• scripts/verify_run.sh — re-runs a worktree's suite under coverage for verification; produced reports/VERIFICATION.md (caught the httpx/iter20 git-restore issue).
• reports/VERIFICATION.md — late-stage verification pass on all 9 coverage-driven control runs.
• .gitignore — excludes the OSS source directories, venvs, __pycache__, etc.

Changed (vs the coverage-driven control's embedded prompts)

• Goal: coverage% → scorecard win. Sessions previously stopped at coverage parity; now they iterate until the scorecard can't improve.
• iter20 stopping condition: was "exceed baseline line+branch %"; is now "3 consecutive iterations without scorecard improvement OR 20 iterations reached."
• iter2 stopping condition: iter_2 must explicitly deepen on scorecard axes, not just close coverage holes.
• SUMMARY format: now reports pure line %, pure branch %, AND combined % side-by-side. (The coverage-driven control SUMMARYs labeled inconsistently — see Finding 8c.)
• Mock-LOC metric: split into mock_real_loc (unittest.mock/MagicMock/mocker — quality concern) and mock_framework_loc (httpx.MockTransport/pytest-httpbin/monkeypatch — legitimate framework use).
• Each iteration's commit message: must name its scorecard move (a)–(f) and the axis delta. Makes coverage-chasing visible in git log.
• Worktree naming: the quality experiment uses wt-r2-<policy> so the coverage-driven control's wt-<policy> stays preserved.
• Branch naming: the quality experiment uses rex-r2-wt-<policy> so the coverage-driven control's rex-wt-<policy> stays preserved.

Hard constraints added (coverage-driven control vulnerabilities)

• No git-history recovery of deleted tests. The coverage-driven control's httpx/wt-iter20 restored 31 of 32 baseline files via git show <delete-commit>^:tests/.... Explicit DO NOT git show / log -p / restore --source / read baseline test bodies clauses in common_header.md.
• No pip install -e . from inside worktrees. The coverage-driven control's requests/base/.venv editable install was clobbered to point at wt-iter20/src/. Prompt now states the shared venv is sufficient.
• REPL verification required for any test that asserts on stdlib / third-party runtime behavior. The coverage-driven control's itsdangerous/wt-oneshot committed 2 failing tests that misread urlsafe_b64decode behavior; both would have been caught by a one-line REPL check.

Tooling fixes

• scripts/setup_quality.py includes quality_scorecard.md in composed prompts (the original draft omitted it; fixed before any quality worktree is materialized).
• scripts/aggregate_results.py src-prefix filter expected relative paths; coverage JSONs store either relative or absolute keys (absolute when the package resolves via an editable install). Fixed: now matches on a normalized path segment (_matches_src/_is_test_file), so both forms count. Re-running --experiment coverage recovers the verified coverage figures (itsdangerous + requests/oneshot+iter2 were silently reporting 0.00 %).
• scripts/aggregate_results.py mock metric split into mock_real_loc (MagicMock|Mock(|patch(|mocker|unittest.mock) and mock_framework_loc (MockTransport|WSGITransport|ASGITransport|monkeypatch|httpbin), per Finding 6.
• scripts/aggregate_results.py parameterized by --experiment <name> (default quality → targets wt-r2-*); writes results-quality.{json,md} so the coverage-driven control's preserved results.{json,md} are never clobbered.
• Still pending: the aggregator reads coverage only. It does not yet read the per-worktree final_scorecard.json to produce a cross-worktree scorecard tally (the actual quality goal). That reader needs a real final_scorecard.json sample to fix its schema against — to be added once the first quality session emits one.

Falsifiable predictions for the quality experiment

See prompts/quality/README.md. Headline predictions:

• Substring-match assertions drop by >80 % vs the coverage-driven control.
• Private-symbol imports go to zero.
• No worktree restores baseline tests from git.
• iter20 iterations used rises from the coverage-driven control's 2–3 toward the 20 budget.
• Final scorecard tally is positive for ≥5 of 9 worktrees.

Ablation arm — Opus 4.7 + xhigh (added 2026-05-28, post-execution)

The coverage-driven control (Opus 4.7, old prompts) → the quality experiment (Opus 4.8, new prompts) confounds the model change with the prompt change. To decompose them we added an ablation arm that holds the prompts fixed and reverts only the model:

• Worktrees: wt-r2b-<policy> on branches rex-r2b-wt-<policy> (9 arms).
• Prompts: byte-identical to the wt-r2-* (quality) prompts except the substituted worktree path — verified by diff. Same instrument.
• Model: claude-opus-4-7, --effort xhigh (matches the quality experiment's 4.8 xhigh; the global effortLevel: xhigh setting applied to both).
• The two comparisons this enables:
  • quality (4.8) vs ablation (4.7), prompts fixed → model effect.
  • ablation (4.7, new prompts) vs the coverage-driven control (4.7, old prompts) → prompt effect. Confirmed clean: the coverage-driven control also ran Opus 4.7 at xhigh (xhigh was the default effortLevel then too), so model + effort are held fixed across this leg.

So the three points form a clean decomposition — the coverage-driven control (4.7·xhigh·coverage-prompts) → ablation (4.7·xhigh·scorecard-prompts) isolates the prompt change; ablation → quality (4.8·xhigh·scorecard-prompts) isolates the model change. No remaining confound.

Tooling (added for quality execution + analysis)

• scripts/setup_quality.py parameterized: --experiment (worktree/branch prefix), --model, --effort. Defaults reproduce the original 4.8 quality behavior; --experiment ablation --model claude-opus-4-7 --effort xhigh built the control.
• scripts/launch_quality.sh takes an optional experiment arg (default quality); launch_quality.sh ablation.
• scripts/score_quality.py (new): independent recompute of the auto-countable scorecard axes (A.1/A.2/A.4/A.5/C.1/B.1/D.1/D.2) applied uniformly to all generated suites + baselines — sidesteps the mutually-incompatible self-reported scorecard.json shapes each session emitted. --experiment.
• scripts/aggregate_results.py (coverage) gained --experiment; writes results-<tag>.{json,md}.

---

[Coverage-driven control] — 2026-05-28 (executed and preserved)

Initial benchmark run. Three repos × three policies = 9 worktree sessions on Opus 4.7. Coverage was the success criterion.

Setup

• Cloned pallets/itsdangerous, encode/httpx, psf/requests into <repo>/base/ (shallow clones).
• Created Python 3.12 venv per repo at <repo>/base/.venv, installed test deps per the project's requirements*.txt / pip install -e ..
• Unified coverage config (bench.coveragerc) per repo: source = <package>, branch = True.
• 3 git worktrees per repo on branches rex-wt-{oneshot,iter2,iter20}.

Initial prompt design (embedded in plan doc)

• Common header: paths, env (. ../base/.venv/bin/activate), bench.coveragerc, baseline figures, repo-specific pytest extra flag (-p no:unraisableexception for httpx).
• Per-policy bodies:
  • oneshot: one generation pass, no repair; commit failures as-is.
  • iter2: up to 2 generate→run→fix iterations.
  • iter20: up to 20 iterations; stop early when baseline coverage is beaten.

Tooling

• scripts/gen_prompts.py — wrote .rex_prompt.md per worktree.
• scripts/launch_all.sh — opened 9 iTerm Claude tabs (--permission-mode auto --model claude-opus-4-7).
• scripts/summarize_coverage.py — extract pure stats from coverage JSON.
• scripts/aggregate_results.py — cross-worktree summary table.
• Later additions during run:
  • scripts/verify_run.sh — re-run a worktree's suite for verification.
  • 9 per-run reports in reports/.
  • audit_itsdangerous.md — deep 3-agent fragility audit.
  • reports/VERIFICATION.md — late-stage integrity check.

Outcomes

9 SUMMARYs + per-run reports in reports/. Net verdicts:

Run	Overall
itsdangerous / oneshot	Worse
itsdangerous / iter2	Worse
itsdangerous / iter20	Worse
httpx / oneshot	Worse
httpx / iter2	Worse
httpx / iter20	Not legitimate — git-restored baseline
requests / oneshot	Worse
requests / iter2	Slightly better
requests / iter20	Better

Only requests/iter20 is unambiguously better than its baseline, via real-I/O integration tests (pytest-httpbin) with zero mock LOC.

See reports/README.md and reports/VERIFICATION.md for details.