Skip to content

Fix benchmark noise status reporting#324

Merged
justrach merged 1 commit intorelease/0.2.579from
fix/bench-report-noise-status
Apr 26, 2026
Merged

Fix benchmark noise status reporting#324
justrach merged 1 commit intorelease/0.2.579from
fix/bench-report-noise-status

Conversation

@justrach
Copy link
Copy Markdown
Owner

Summary

  • make benchmark markdown status match the actual CI gate: percentage threshold plus absolute ns threshold
  • add an absolute delta column so tiny-tool percentage swings are easier to read
  • label high-percent, low-absolute rows as NOISE instead of FAIL
  • add Python unit coverage for the status/report rendering

Validation

  • python3 -m unittest scripts/test_compare_bench.py
  • python3 -m py_compile scripts/compare-bench.py scripts/test_compare_bench.py
  • manual sample: codedb_read +22.54% / +11,399 ns now renders as NOISE and exits 0

@github-actions
Copy link
Copy Markdown

Benchmark Regression Report

Thresholds: 10.00% and 50,000 ns absolute delta

NOISE means the percentage threshold was exceeded, but the absolute delta was too small to fail CI.

Tool Base (ns) Head (ns) Delta Abs Delta (ns) Status
codedb_bundle 306489 308775 +0.75% +2286 OK
codedb_changes 34183 30193 -11.67% -3990 OK
codedb_deps 5139 5314 +3.41% +175 OK
codedb_edit 4200 4766 +13.48% +566 NOISE
codedb_find 42827 41409 -3.31% -1418 OK
codedb_hot 57812 56980 -1.44% -832 OK
codedb_outline 155173 158092 +1.88% +2919 OK
codedb_read 53074 60066 +13.17% +6992 NOISE
codedb_search 146206 144245 -1.34% -1961 OK
codedb_snapshot 2149126 2142620 -0.30% -6506 OK
codedb_status 242009 237537 -1.85% -4472 OK
codedb_symbol 43152 36751 -14.83% -6401 OK
codedb_tree 41413 40454 -2.32% -959 OK
codedb_word 42898 38859 -9.42% -4039 OK

@justrach justrach merged commit 07ac438 into release/0.2.579 Apr 26, 2026
1 check passed
@justrach justrach deleted the fix/bench-report-noise-status branch April 26, 2026 01:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant