sloppoke-bench

Public, reproducible measurement of how good sloppoke is at catching AI-introduced slop that human review missed. One script, one number, one published claim.

When the CI badge above is green: every unit test passes AND sloppoke.me's commercial API has returned the canonical numbers on the latest push to dev.

The headline result from this repo:

Sloppoke's pre-commit gate would have flagged 39.9% of the static-analysis issues that AI commits shipped past human review and reached HEAD in 5,173 production OSS repos.

53.0% of surviving WARNING-class findings. 84.7% of surviving REFACTOR-class smells. 25.7% of surviving CONVENTION-class. See results/ for the full table.

What lives here

crawler/analysis.py            — the script that computes the headline
crawler/tests/test_analysis.py — 20 unit tests against hand-checked numbers
results/                       — the canonical output anyone re-running
                                 the script can diff against
README.md, LICENSE, pyproject.toml

If a script doesn't produce a number sloppoke.me publishes, it isn't here.

How to run it

# 1. Download the source archive (Liu et al. 2026, AI-TD)
mkdir -p ai-td && cd ai-td
gdown --folder 'https://drive.google.com/drive/folders/1R3uiO6dt9gWEq_2njwS-ve4DawxvpsS1'
cd tech-debt-ai-coding-data && unzip commits.zip && unzip results-out.zip
cd ../..

# 2. Run the effectiveness analysis
python -m sloppoke_bench.crawler.analysis effectiveness \
    --ai-td-root ai-td/tech-debt-ai-coding-data

Output: crawler/output/analysis_sloppoke_effectiveness_<utc>.{json,md}.

Diff against results/analysis_sloppoke_effectiveness_*.json to verify your re-run matches the canonical artefact.

Source dataset

dataset	scope	URL
AI-Introduced Technical Debt (Liu et al. 2026, arXiv:2603.28592)	304K verified AI-authored commits / 6,299 repos / 5 commercial agents + 9 long-tail tools	github.com/yueyueL/tech-debt-ai-coding + Google Drive mirror

Tests

python -m venv .venv && source .venv/bin/activate
pip install -e .[test]
pytest -q

20 tests cover the catalog→rule mapping logic, the survival aggregation, the severity breakdown, the markdown renderer, and a custom-covers injection so a reviewer can audit the headline under a different mapping. CI runs them on every push.

If a number sloppoke.me cites doesn't trace to a command in this repo, ask.

License

MIT. See LICENSE.