Add CI, Kotlin/Swift dashboard, and repo-consistency cleanups - #4
Merged
Conversation
- Add a GitHub Actions workflow running all three pytest suites (benchmark tooling + both bundled skills) plus the sabotage check. - Generate the missing Kotlin/Swift interactive dashboard (docs/kotlin-swift-results.html) from the committed scorecard, with a static preview in examples/ and README links alongside the other two. - Drop the Kotlin/Swift repos from score_cross_language.py: the committed matrix was scored by the manifest-driven, module-scoped score_kotlin_swift_matrix.py, and rescoring them here (whole-repo, unscoped) would merge contradictory baselines into the cross-language scorecard. - Correct the CHANGELOG claim that the C.1 instruments "now agree" — score.py and aggregate_results.py still match different textual forms of the stdlib mock.patch idiom. - Frame reports/ as the coverage-driven control up front, so its mostly-"Worse" verdict matrix reads as the motivation for the quality experiment rather than a contradiction of it. - Move the orphaned baseline_summary.md into reports/ and link it; clarify README's 185→59 A.2 claim (coverage-arm vs quality-arm, not vs baseline); fix SKILL.md's two stale python|js|go-only mentions; mark gen_prompts.py and verify_run.sh as historical coverage-control tooling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011BDzS3SyE6Sh3GE4nnSoWN
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
An audit-driven cleanup pass — the highest-value gaps found in a full repo review:
.github/workflows/ci.yml): the repo's thesis is regression-catching test quality, but nothing ran its own tests. The workflow runs all three pytest suites (benchmark tooling,test-qualityskill,results-dashboardskill — 146 tests) plusscripts/sabotage_check.py(9/9 curated mutations) on every push/PR.docs/kotlin-swift-results.html+examples/kotlin-swift-results.png): the most-promoted result (all 18 arms beat baseline) was the only one without an interactive dashboard. Generated from the committed scorecard with the bundledresults-dashboardskill; linked from the README alongside the Python and JS/Go dashboards.score_cross_language.pystill listed the six Kotlin/Swift repos unscoped (tests_dir="."), but the committed matrix was scored by the manifest-driven, module-scopedscore_kotlin_swift_matrix.py. Rescoring them via the cross-language script would merge contradictory baselines into that scorecard. Dropped, with a comment pointing at the real scorer.score.pycatchesmock.patch(...)via amock.patchalternative whileaggregate_results.pymatches the literalunittest.mock(which doesn't fire onfrom unittest import mock). Reworded to state the remaining divergence.reports/framing: the coverage-driven control's index showed a mostly-"Worse" verdict matrix with no hint it is the control — a reader landing there saw the tool losing 8 of 9 with nothing tying it to the 9/9 quality result. Added a framing note up top.baseline_summary.mdintoreports/and linked it; clarified README's185 → 59A.2 claim (coverage-arm vs quality-arm totals, not vs baseline); fixedSKILL.md's two stalepython|js|go-only mentions (the skill supports Kotlin + Swift); markedgen_prompts.py/verify_run.shas historical coverage-control tooling.Verification
pytest tests/ .claude/skills/test-quality/tests/ .claude/skills/results-dashboard/tests/→ 146 passed, 4 xfailed (the exact CI invocation)python scripts/sabotage_check.py→ 9/9 mutations caught🤖 Generated with Claude Code
https://claude.ai/code/session_011BDzS3SyE6Sh3GE4nnSoWN